Rich Sutton has prepared a FAQ about reinforcement learning — how to think\u00a0about it and how do it successfully in practice. This FAQ is an\u00a0attempt to pull together in one place some of the more common\u00a0questions and answers.<\/p>\n
“I have been free with my opinions, but I\u00a0would also welcome the opinions of others for inclusion here. In you\u00a0have anything to add to my comments, including whole new questions\u00a0or totally different answers, please send me a note at\u00a0rich@richsutton.com.” – Rich Sutton<\/p>\n
General Questions<\/a><\/p>\n <\/a><\/p>\n <\/a><\/p>\n <\/a><\/p>\n <\/a><\/p>\n <\/a><\/p>\n <\/a><\/p>\n <\/a><\/p>\n <\/a><\/p>\n <\/a><\/p>\n <\/a>Questions about studying and teaching RL<\/a><\/p>\n Questions about the RL textbook by Sutton and Barto<\/a>?<\/p>\n <\/a><\/p>\n <\/a><\/p>\n <\/a><\/p>\n <\/a><\/p>\n Nuts and Bolts of RL<\/a><\/p>\n <\/a><\/p>\n Advice and Opinions<\/a><\/p>\n <\/a><\/p>\n Miscellaneous Questions<\/a><\/p>\n <\/a><\/p>\n General Questions<\/strong><\/p>\n <\/a><\/p>\n What is Reinforcement Learning?<\/strong><\/p>\n Reinforcement learning (RL) is learning from interaction with an\u00a0environment, from the consequences of action, rather than\u00a0from explicit teaching. RL become popular in the 1990s within machine learning and artificial intelligence,but also within operations research and with offshoots in psychology and neuroscience.<\/p>\n Most RL research is conducted within the mathematical framework of\u00a0Markov decision processes (MDPs). MDPs involve a decision-making agent interacting with its environment so as to maximize the cumulative reward it receives over time. The agent perceives aspects of the environment’s state and selects actions. The agent may estimate a value function and use it to construct better and better decision-making policies over time.<\/p>\n RL algorithms are methods for solving this kind of problem, that is, problems involving sequences of decisions in which each decision affects what opportunities are available later, in which the effects need not be\u00a0deterministic, and in which there are long-term goals. RL methods are intended to address the kind of learning and decision making problems that people and animals face in their normal, everyday lives.<\/p>\n For more information, see the sources recommended for an introduction to RL<\/a>.<\/p>\n <\/a><\/p>\n Is RL just trial-and-error learning, or does it include planning?<\/strong><\/p>\n Modern reinforcement learning concerns both trial-and-error learning\u00a0without a model of the environment, and deliberative planning with a\u00a0model. By “a model” here we mean a model of the dynamics of the\u00a0environment. In the simplest case, this means just an estimate of the\u00a0state-transition probabilities and expected immediate rewards of the\u00a0environment. In general it means any predictions about the\u00a0environment’s future behavior conditional on the agent’s behavior.<\/p>\n <\/a><\/p>\n How does RL relate to Neuro-Dynamic Programming?<\/strong><\/p>\n To a first approximation, Reinforcement Learning and Neuro-Dynamic\u00a0Programming are synonomous. In fact, neither name is very descriptive of the subject, and I\u00a0recommend you use neither when you want to be technically precise.\u00a0Names such as this are useful when referring to a general The problem with “reinforcement learning” is that it is dated. Much\u00a0of the field does not concern learning at all, but just planning\u00a0from complete knowledge of the problem (a model of the environment).\u00a0Almost the same methods are used for planning and learning, and it seems\u00a0off-point to emphasize learning in the name of the field.\u00a0“Reinforcement” does not seem particularly pertinent either.<\/p>\n The problem with “neuro-dynamic programming” is similar in that neither\u00a0neural networks nor dynamic programming are critical to\u00a0the field. Many of the methods, such as “Monte Carlo”, or “rollout” methods,\u00a0are completely unrelated to dynamic programming, and neural networks\u00a0are just one choice among many for method of function approximation.\u00a0Moreover, one could argue that the component names, “neural networks”\u00a0and “dynamic programming”, are each not very descriptive of their respective\u00a0methods.<\/p>\n <\/a><\/p>\n What advantages does RL offer in Operations Research problems?<\/strong><\/p>\n Using function approximation, RL can apply to much larger\u00a0state spaces than classical sequential optimization techniques such\u00a0as dynamic programming. In addition, using simulations (sampling), RL\u00a0can apply to systems that are too large or complicated to explicitly\u00a0enumerate the next-state transition probabilities.<\/p>\n <\/a><\/p>\n How does RL relate to Neuroscience?<\/strong><\/p>\n Ideally, the ideas of reinforcement learning could constitute part of a\u00a0computational theory<\/em> of what the brain is doing and why. A number\u00a0of links have been drawn between reinforcement learning and neuroscience,\u00a0beginning with early models of classical conditioning based on\u00a0temporal-difference learning (see\u00a0Barto and Sutton, 1982<\/a>;\u00a0Sutton and Barto, 1981<\/a>,\u00a01990<\/a>;\u00a0Moore et al., 1986<\/a>), <\/a><\/p>\n How does RL relate to the psychology of animal behavior?<\/strong><\/p>\n Broadly speaking, RL works as a pretty good model of instrumental learning,\u00a0though a detailed argument for this has never been publically made (the closest to\u00a0this is probably Barto, Sutton and Watkins, 1990<\/a>). \u00a0On the other hand, the links between classical (or Pavlovian) conditioning and\u00a0temporal-difference (TD) learning (one of the central elements of RL) are close and\u00a0widely acknowledged (see Sutton and Barto, 1990<\/a>). \u00a0Ron Sun<\/a> has developed hybrid models combining high-level and low-level\u00a0skill learning, based in part on RL, which make contact with\u00a0psychological data (see \u00a0Sun, Merrill, and Peterson, 2001<\/a>).<\/p>\n <\/a><\/p>\n How does RL relate to behaviorism?<\/strong><\/p>\n Formally, RL is unrelated to behaviorism, or at least to the aspects\u00a0of behaviorism that are widely viewed as undesireable. Behaviorism\u00a0has been disparaged for focusing exclusively on behavior, refusing to consider\u00a0what was going on inside the head of the subject. RL of course is all about the\u00a0algorithms and processes going on inside the agent. For example, we often consider Nevertheless, RL shares with behaviorism its origins in animal learning theory, and\u00a0in its focus on the interface with the environment. RL’s states and actions are\u00a0essentially animal learning theory’s stimuli and responses. Part of RL’s point is\u00a0that these are the essential common final path for all that goes on in the agent’s\u00a0head. In the end it all comes down to the actions taken and the states perceived.<\/p>\n <\/a><\/p>\n What are some of the more significant applications of RL?<\/strong><\/p>\n <\/a><\/p>\n How does RL relate to genetic algorithms?<\/strong><\/p>\n Most work with genetic algorithms simulates evolution, not learning during\u00a0an individual’s life, and because of this is very different from work in RL. That having\u00a0been said, there are two provisos. First, there is a large body of work on\u00a0classifier systems that uses or is closely related to genetic algorithms.\u00a0This work is concerned with learning during a single agent’s lifetime\u00a0(using GAs to organize the components of the agent’s mind) and is in fact\u00a0RL research. The second proviso is that GA work is\u00a0often related to RL by virtue of being applied to the same problems<\/em>.\u00a0For example, GA methods can be applied to evolve<\/em> a backgammon player\u00a0and that player can be compared with a player learned by RL methods. In\u00a0fact, a large portion of systems evolved by GAs are controllers that could\u00a0alternatively be learned by RL methods. It is tempting here to make a\u00a0blanket statement about which class of methods is more appropriate or\u00a0performs better. A crucial distinction is that between problems in\u00a0which state information is available and those in which it is not. In\u00a0backgammon, for example, the state of the board is available. In problems like these RL methods seem to\u00a0have a distinct advantage over evolutionary methods such as GAs. On other\u00a0problems there is little state information. Suppose you wish to learn a\u00a0golf swing, or some other ballistic motion that is generally over before\u00a0useful sensory feedback is available. Then the system is far from Markov and\u00a0there is little or no advantage provided by addressing the state\u00a0information during a single swing. In these cases RL methods would not be\u00a0expected to have an advantage over evolutionary methods. In fact, if they\u00a0insisted on trying to treat sensory information as state, as in using\u00a0temporal-difference methods, they may well do worse.<\/p>\n <\/a><\/p>\n Why don’t you include genetic algorithms in RL?<\/strong><\/p>\n <\/a><\/p>\n Who invented RL?<\/strong><\/p>\n There is plenty of blame here to go around.\u00a0Minsky, Bellman, Howard, Andreae, Werbos, Barto, Sutton, Watkins… All played\u00a0important or early roles. See the\u00a0history from the Sutton and Barto text<\/a>.<\/p>\n <\/a><\/p>\n Questions about studying and teaching RL<\/strong><\/p>\n <\/a><\/p>\n What sources do you recommend for an introduction to <\/strong>reinforcement learning?<\/strong><\/p>\n For a general introduction, I recommend my book with Prof. Barto:\u00a0Reinforcement Learning: An Introduction,\u00a0by Richard S. Sutton and Andrew G. Barto. MIT Press 1998.\u00a0Online version<\/a>.\u00a0There is also a Japanese translation available. \u00a0For a more formal treatment, including rigorous proofs, I recommend the\u00a0text by Bertsekas and Tsitsiklis:\u00a0Neuro-dynamic Programming<\/a>,\u00a0by Dimitri P. Bersekas and John N. Tsitsiklis.\u00a0Athena Press, 1996. \u00a0If you don’t have time for a textbook-length treatment, your best\u00a0bet is one or both of these two papers:\u00a0Reinforcement learning: A survey<\/a>, by Kaelbling, L.P., Littman, M.L.,\u00a0and Moore, A.W., in the Journal of Artificial Intelligence <\/em>Research<\/em>, 4:237–285, 1996. \u00a0Learning and sequential decision making<\/a>, by\u00a0Barto, A.G., Sutton, R.S., & Watkins, C.J.C.H.,\u00a0in Learning and Computational Neuroscience<\/em>,\u00a0M. Gabriel and J.W. Moore (Eds.), pp. 539–602, 1990, MIT Press.<\/p>\n <\/a><\/p>\n Questions about the RL textbook by Sutton and Barto<\/strong><\/p>\n <\/a><\/p>\n Where can I find an online version of the Sutton and Barto book?<\/strong><\/p>\n http:\/\/richsutton.com\/book\/the-book.html<\/a><\/p>\n There is also a lot of other material there: code, figures, errata,some teaching materials.<\/p>\n <\/a><\/p>\n Where can I find an electronic version of the book suitable for printing?<\/strong><\/p>\n Pdfs of pre-publication versions of Chapters 1, 3, and 9 (only) are available at\u00a0http:\/\/richsutton.com\/book\/the-book.html<\/a><\/p>\n <\/a><\/p>\n Where can I find the answers to the exercises?<\/strong><\/p>\n An instructor’s manual containing answers to all the non-programming\u00a0exercises is available to qualified teachers. Readers using the book for self study can obtain answers on a\u00a0chapter-by-chapter basis after working on the exercises themselves. \u00a0For details see here<\/a>.<\/p>\n <\/a><\/p>\n I have found an apparent error in the book. What should I do about it?<\/strong><\/p>\n First, please check the list of errata<\/a>. If you don’t find your error\u00a0there, then please send me a note describing it at rich@richsutton.com. \u00a0If I agree that it is an\u00a0error, then I will add it the the errata list. For your troubles you will\u00a0be credited in the list for the discovery of the error.<\/p>\n <\/a><\/p>\n Where can I find some demos of RL?<\/strong><\/p>\n <\/a><\/p>\n What are some important open problems in RL?<\/strong><\/p>\n <\/a><\/p>\n How can I obtain an evaluation copy of the book?<\/strong><\/p>\n Qualified instructor can generally obtain an examination copy from\u00a0MIT press. Please see the \u00a0MIT Press home page for the book<\/a>.<\/p>\n\n
\n
\n
\n
\n
\n
\nwhat groups of them to treat conjunctively, how many tilings, the
\ntile widths, etc? Do we have to make all these kind of representational
\ndecisions ourselves, as system designers, or are there ways of
\nautomating these decisions?<\/a><\/li>\n\n
\n
\n
\nthese are limited. Why not use the powerful nonlinear function
\napproximators such as neural networks, decision trees, and so forth
\nthat are so popular in supervised learning?<\/a><\/li>\n\n
\n
\n
\n
\n
\nThe name “reinforcement learning” came from psychology (although\u00a0psychologists rarely use exactly this term) and dates back to the eary days of cybernetics. For example, Marvin Minsky used this term in his 1954 thesis, and Barto and Sutton revived it in the early 1980’s. The name “neuro-dynamic programming” was coined by Bertsekas and Tsitsiklis in 1996 to capture the idea of the field as a combination of neural networks and dynamic programming.<\/p>\n
\nbody of research, but not for carefully distinguishing ideas\u00a0from one another. In that sense, there is no point in trying\u00a0to draw a careful distinction between the referents of these two names.<\/p>\n
\n
\n
\nand continuing through work on foraging\u00a0and prediction learning (see\u00a0Montague et al., 1995<\/a>,\u00a01996<\/a>),\u00a0and on dopamine neurons in monkeys as a temporal-difference-error\u00a0distribution system. A good\u00a0survey paper<\/a> is available.\u00a0See also Suri, submitted<\/a>. \u00a0A\u00a0book<\/a> collects a number of\u00a0relevant papers. Doya has extensively developed\u00a0RL models of the basal ganglia<\/a>.\u00a0Many of these areas are very active at present and changing rapidly.<\/p>\n
\n
\n
\nthe construction of internal models of the environment within the agent, which is\u00a0far outside the scope of behaviorism.<\/p>\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n