Inside the LLM Post-Training Knowledge Graph: A Systematic Deep-Dive into Reasoning Models
Hook
While developers scramble to implement RLHF in production, the real bottleneck isn’t code—it’s knowing which of the dozens of different post-training techniques published recently actually matter.
Context
The LLM post-training landscape has exploded into a fragmented maze of techniques: Direct Preference Optimization (DPO), Proximal Policy Optimization (PPO), Constitutional AI, reward modeling, test-time compute scaling, and dozens more. Each promises to make models “reason better,” but the field moves so fast that even full-time researchers struggle to maintain a mental model of what exists, what works, and how techniques relate to each other.
The Awesome-LLM-Post-training repository from MBZUAI’s Oryx team addresses this chaos by constructing a systematic taxonomy of post-training methods. Unlike typical “awesome lists” that dump links chronologically, this repository is architecturally grounded in a formal survey paper (arXiv:2502.21321) that categorizes the field into three major pillars: fine-tuning approaches, reinforcement learning methods, and test-time scaling techniques. It’s essentially a navigable knowledge graph of an extensive collection of papers, explicitly designed to help researchers understand not just what exists, but how different approaches connect conceptually—from classical RL theory through modern preference learning to emerging System 2 reasoning paradigms.
Technical Insight
The repository’s true value lies in its taxonomic structure, which mirrors the theoretical framework laid out in the accompanying survey paper. The main README organizes content into multiple hierarchical sections, each representing a distinct conceptual cluster in the post-training design space. The top-level categories—Reward Learning, Policy Optimization, LLMs for Reasoning & Decision-Making—aren’t arbitrary; they reflect fundamental architectural choices in how you shape model behavior post-pretraining.
Consider the Reward Learning section, which breaks down into three subsections: Human Feedback (RLHF variants), Preference-Based RL (DPO, IPO, KTO), and Intrinsic Motivation (curiosity-driven approaches). This organization reveals a key insight: you can’t just “do RLHF.” You need to decide whether you’re learning an explicit reward model (RLHF), directly optimizing preferences without a reward model (DPO), or bootstrapping rewards from model self-evaluation. The repository links each approach to foundational papers, making it easy to trace intellectual lineage. For example, the DPO paper explicitly positions itself as removing the reward model training phase from RLHF, and the taxonomy makes this relationship immediately visible.
The Policy Optimization section demonstrates similar depth, covering Offline RL (training from fixed datasets without environment interaction), Imitation Learning (behavioral cloning from expert demonstrations), and Hierarchical RL (decomposing complex tasks into subgoals). These aren’t just alternative techniques—they represent fundamentally different assumptions about your training setup. If you’re training a code generation model from GitHub data, you’re inherently in an offline RL regime where you can’t query the “environment” for new trajectories. The repository doesn’t just list papers; it contextualizes when each approach applies.
What makes this resource particularly valuable for practitioners is the Applications & Benchmarks section, which bridges theory and evaluation. It links to concrete benchmarks like GSM8K (math reasoning), HumanEval (code generation), and MMLU (general knowledge), alongside papers describing how models were evaluated. This matters because post-training technique choice is often benchmark-dependent—methods that excel at multi-step mathematical reasoning (like process reward models with step-level supervision) may underperform on open-ended creative tasks where defining “correctness” is ambiguous.
The repository also tracks the emerging frontier of test-time compute scaling, documented in its own dedicated section. Unlike traditional post-training that modifies model weights, test-time scaling techniques like Chain-of-Thought, Tree-of-Thoughts, and self-consistency sampling allocate more inference compute to improve reasoning without retraining. The taxonomy positions this as a distinct paradigm shift: moving from “better training” to “better inference.” Papers on techniques like Best-of-N sampling and verification-guided decoding show up here, revealing how modern systems like o1 likely combine extensive post-training with sophisticated inference-time search.
The Multi-Agent RL section is particularly forward-looking, covering emergent communication, coordination mechanisms, and social learning—techniques that matter as we move toward agentic systems that need to collaborate or negotiate. While most current LLM work focuses on single-model performance, this section acknowledges that future applications (multi-agent debate, collaborative problem-solving) will require different optimization objectives entirely.
Gotcha
The repository’s greatest strength—comprehensive breadth—is also its primary limitation for practitioners seeking implementation guidance. While this is a curated collection of research papers with links to original sources, many of which include code repositories, there’s no unified framework or ready-to-run implementations in this repository itself. Most links point to academic papers (arXiv PDFs, conference proceedings), and while many papers include associated code repositories, you’re responsible for finding, vetting, and adapting those implementations yourself. You’re expected to read papers, understand the theoretical foundations, and implement or adapt existing code independently.
The sheer scope creates a second problem: cognitive overwhelm. A newcomer wanting to “learn about RLHF” will find themselves facing numerous related papers spanning reward modeling theory, KL-divergence regularization techniques, online vs. offline variants, and hybrid approaches. Without existing domain knowledge, it’s hard to know where to start or which papers are foundational versus incremental. The repository provides taxonomic structure but doesn’t offer explicit learning paths or recommended reading orders beyond its categorical organization. You need some baseline understanding of the field’s conceptual landscape to navigate efficiently—exactly the knowledge the repository could help build, creating a chicken-and-egg problem.
Maintenance is another consideration. The LLM research field publishes hundreds of papers monthly, and the repository already includes papers through early 2025. Sustaining this curation rate requires significant ongoing effort, and there’s no automated pipeline for discovering, categorizing, and adding new work. As techniques evolve and some approaches become deprecated, the risk of outdated or incomplete coverage grows. The repository currently shows 2,342 stars (as of collection date), suggesting strong community engagement, but long-term maintenance sustainability depends on continued contributor activity.
Verdict
Use this repository if you’re conducting a literature review for a research paper, thesis, or grant proposal on LLM capabilities; if you’re a machine learning engineer tasked with evaluating multiple post-training approaches and need to quickly understand the design space; if you’re an AI researcher trying to position your work relative to existing techniques; or if you’re building a course curriculum on modern LLM training methods and need a comprehensive, organized bibliography. It’s one of the most comprehensive single-source overviews of post-training techniques available, backed by rigorous academic categorization and grounded in a formal survey paper. Skip it if you need production-ready code libraries with installation instructions and tutorials—you’ll need to explore the individual paper repositories linked within; if you’re a complete beginner wanting a tutorial-style introduction to RLHF with worked examples—consider starting with introductory materials before diving into this research-focused collection; if you need narrow, deep expertise on one specific technique rather than broad survey knowledge; or if you prefer video content or interactive learning over reading academic papers. This is a researcher’s tool for mapping the territory, not an engineer’s toolkit with ready-to-deploy implementations.
