 # Epsilon-Greedy Algorithm in Reinforcement Learning

## Epsilon-Greedy Algorithm in Reinforcement Learning

In Reinforcement Learning, the agent or decision-maker learns what to do—how to map situations to actions—so as to maximize a numerical reward signal. The agent is not explicitly told which actions to take, but instead must discover which action yields the most reward through trial and error.

Multi-Armed Bandit Problem

The multi-armed bandit problem is used in reinforcement learning to formalize the notion of decision-making under uncertainty. In a multi-armed bandit problem, an agent(learner) chooses between k different actions and receives a reward based on the chosen action.
The multi-armed bandits are also used to describe fundamental concepts in reinforcement learning, such as rewardstimesteps, and values.
For selecting an action by an agent, we assume that each action has a separate distribution of rewards and there is at least one action that generates maximum numerical reward. Thus, the probability distribution of the rewards corresponding to each action is different and is unknown to the agent(decision-maker). Hence, the goal of the agent is to identify which action to choose to get the maximum reward after a given set of trials.

Action-Value and Action-Value Estimate
For an agent to decide which action yields the maximum reward, we must define the value of taking each action. We use the concept of probability to define these values using the action-value function.
The value of selecting an action is defined as the expected reward received when taking that action from a set of all possible actions. Since the value of selecting an action is not known to the agent, so we use the ‘sample-average’ method to estimate the value of taking an action. Exploration vs Exploitation

Exploration allows an agent to improve its current knowledge about each action, hopefully leading to long-term benefit. Improving the accuracy of the estimated action-values, enables an agent to make more informed decisions in the future.

Exploitation on the other hand, chooses the greedy action to get the most reward by exploiting the agent’s current action-value estimates. But by being greedy with respect to action-value estimates, may not actually get the most reward and lead to sub-optimal behaviour.
When an agent explores, it gets more accurate estimates of action-values. And when it exploits, it might get more reward. It cannot, however, choose to do both simultaneously, which is also called the exploration-exploitation dilemma.

Epsilon-Greedy Action Selection
Epsilon-Greedy is a simple method to balance exploration and exploitation by choosing between exploration and exploitation randomly.
The epsilon-greedy, where epsilon refers to the probability of choosing to explore, exploits most of the time with a small chance of exploring.  Code: Python code for Epsilon-Greedy

 `# Import required libraries` `import` `numpy as np` `import` `matplotlib.pyplot as plt` ` ` `# Define Action class` `class` `Actions:` `  ``def` `__init__(``self``, m):` `    ``self``.m ``=` `m` `    ``self``.mean ``=` `0` `    ``self``.N ``=` `0` ` ` `  ``# Choose a random action` `  ``def` `choose(``self``): ` `    ``return` `np.random.randn() ``+` `self``.m` ` ` `  ``# Update the action-value estimate` `  ``def` `update(``self``, x):` `    ``self``.N ``+``=` `1` `    ``self``.mean ``=` `(``1` `-` `1.0` `/` `self``.N)``*``self``.mean ``+` `1.0` `/` `self``.N ``*` `x` ` ` ` ` `def` `run_experiment(m1, m2, m3, eps, N):` `     ` `  ``actions ``=` `[Actions(m1), Actions(m2), Actions(m3)]` ` ` `  ``data ``=` `np.empty(N)` `   ` `  ``for` `i ``in` `range``(N):` `    ``# epsilon greedy` `    ``p ``=` `np.random.random()` `    ``if` `p < eps:` `      ``j ``=` `np.random.choice(``3``)` `    ``else``:` `      ``j ``=` `np.argmax([a.mean ``for` `a ``in` `actions])` `    ``x ``=` `actions[j].choose()` `    ``actions[j].update(x)` ` ` `    ``# for the plot` `    ``data[i] ``=` `x` `  ``cumulative_average ``=` `np.cumsum(data) ``/` `(np.arange(N) ``+` `1``)` ` ` `  ``# plot moving average ctr` `  ``plt.plot(cumulative_average)` `  ``plt.plot(np.ones(N)``*``m1)` `  ``plt.plot(np.ones(N)``*``m2)` `  ``plt.plot(np.ones(N)``*``m3)` `  ``plt.xscale(``'log'``)` `  ``plt.show()` ` ` `  ``for` `a ``in` `actions:` `    ``print``(a.mean)` ` ` `  ``return` `cumulative_average` ` ` ` ` `if` `__name__ ``=``=` `'__main__'``:` `     ` `  ``c_1 ``=` `run_experiment(``1.0``, ``2.0``, ``3.0``, ``0.1``, ``100000``)` `  ``c_05 ``=` `run_experiment(``1.0``, ``2.0``, ``3.0``, ``0.05``, ``100000``)` `  ``c_01 ``=` `run_experiment(``1.0``, ``2.0``, ``3.0``, ``0.01``, ``100000``)`

Output: Code: Python code for getting the log output plot

 `# log scale plot` `plt.plot(c_1, label ``=``'eps = 0.1'``)` `plt.plot(c_05, label ``=``'eps = 0.05'``)` `plt.plot(c_01, label ``=``'eps = 0.01'``)` `plt.legend()` `plt.xscale(``'log'``)` `plt.show()`

Output: Code: Python code for getting the linear output plot

 `# linear plot` `plt.figure(figsize ``=` `(``12``, ``8``))` `plt.plot(c_1, label ``=``'eps = 0.1'``)` `plt.plot(c_05, label ``=``'eps = 0.05'``)` `plt.plot(c_01, label ``=``'eps = 0.01'``)` `plt.legend()` `plt.show()`

Output: Last Updated on November 13, 2021 by admin

## Python – Mean Squared ErrorPython – Mean Squared Error

Python | Mean Squared Error The Mean Squared Error (MSE) or Mean Squared Deviation (MSD) of an estimator measures the average of error squares i.e. the average squared difference between the estimated values and true value. It is a risk function, corresponding to the

## K means Clustering – IntroductionK means Clustering – Introduction

K means Clustering – Introduction We are given a data set of items, with certain features, and values for these features (like a vector). The task is to categorize those items into groups. To achieve this, we will use the kMeans

## Naive Bayes ClassifiersNaive Bayes Classifiers

Naive Bayes Classifiers Naive Bayes classifiers are a collection of classification algorithms based on Bayes’ Theorem. It is not a single algorithm but a family of algorithms where all of them share a common principle, i.e. every pair of features being

## Linear RegressionLinear Regression

Machine Learning – Linear Regression Linear Regression is a machine learning algorithm based on supervised learning. It performs a regression task. Regression models a target prediction value based on independent variables. It is mostly used for finding out the relationship between variables and

## Understanding Logistic RegressionUnderstanding Logistic Regression

Understanding Logistic Regression This article discusses the basics of Logistic Regression and its implementation in Python. Logistic regression is basically a supervised classification algorithm. In a classification problem, the target variable(or output), y, can take only discrete values for given

## Logistic Regression using StatsmodelsLogistic Regression using Statsmodels

Logistic Regression using Statsmodels Logistic regression is the type of regression analysis used to find the probability of a certain event occurring. It is the best suited type of regression for cases where we have a categorical dependent variable which

## Random Forest Regression in PythonRandom Forest Regression in Python

Random Forest Regression in Python Every decision tree has high variance, but when we combine all of them together in parallel then the resultant variance is low as each decision tree gets perfectly trained on that particular sample data and

## Label Encoding of datasets in PythonLabel Encoding of datasets in Python

ML | Label Encoding of datasets in Python In machine learning, we usually deal with datasets that contain multiple labels in one or more than one column. These labels can be in the form of words or numbers. To make