boost/math
2
0
Fork 0
mirror of https://github.com/boostorg/math.git synced 2026-02-24 16:12:15 +00:00
Code Issues Projects Releases Wiki Activity

save doc progress

Browse Source
This commit is contained in:
mzhelyez
2025-12-19 08:38:35 +01:00
parent 9e784682fa
commit cdaaeae4a2
7 changed files with 419 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
doc/html/math_toolkit/gd_opt.html Normal file
View File

@@ -0,0 +1,59 @@
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Gradient Based Optimizers</title>
<link rel="stylesheet" href="../math.css" type="text/css">
<meta name="generator" content="DocBook XSL Stylesheets Vsnapshot">
<link rel="home" href="../index.html" title="Math Toolkit 4.2.1">
<link rel="up" href="../optimization.html" title="Chapter 11. Optimization">
<link rel="prev" href="cma_es.html" title="Evolution Strategy with Covariance Matrix Adaptation">
<link rel="next" href="gd_opt/introduction.html" title="Introduction">
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
<table cellpadding="2" width="100%"><tr>
<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../boost.png"></td>
<td align="center"><a href="../../../../../index.html">Home</a></td>
<td align="center"><a href="../../../../../libs/libraries.htm">Libraries</a></td>
<td align="center"><a href="http://www.boost.org/users/people.html">People</a></td>
<td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td>
<td align="center"><a href="../../../../../more/index.htm">More</a></td>
</tr></table>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="cma_es.html"><img src="../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../optimization.html"><img src="../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="gd_opt/introduction.html"><img src="../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
<div class="section">
<div class="titlepage"><div><div><h2 class="title" style="clear: both">
<a name="math_toolkit.gd_opt"></a><a class="link" href="gd_opt.html" title="Gradient Based Optimizers">Gradient Based Optimizers</a>
</h2></div></div></div>
<div class="toc"><dl class="toc">
<dt><span class="section"><a href="gd_opt/introduction.html">Introduction</a></span></dt>
<dt><span class="section"><a href="gd_opt/gradient_descent.html">Gradient Desccent</a></span></dt>
<dt><span class="section"><a href="gd_opt/nesterov.html">Nesterov Gradient Desccent</a></span></dt>
<dt><span class="section"><a href="gd_opt/lbfgs.html">L-BFGS</a></span></dt>
</dl></div>
<p>
Gradient based optimizers are algorithms that use the gradient of a funciton
to iteratively find locally extreme points of functions over a set of parameters.
This sections provides a description of a set of gradient optimizers. The optimizers
are written with <code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">math</span><span class="special">::</span><span class="identifier">differentiation</span><span class="special">::</span><span class="identifier">reverse_mode</span><span class="special">::</span><span class="identifier">rvar</span></code> in
mind, however if a way to evaluate the funciton and its gradient is provided,
the optimizers should work in exactly the same way.
</p>
</div>
<div class="copyright-footer">Copyright © 2006-2021 Nikhar Agrawal, Anton Bikineev, Matthew Borland,
Paul A. Bristow, Marco Guazzone, Christopher Kormanyos, Hubert Holin, Bruno
Lalande, John Maddock, Evan Miller, Jeremy Murphy, Matthew Pulver, Johan Råde,
Gautam Sewani, Benjamin Sobotta, Nicholas Thompson, Thijs van den Berg, Daryle
Walker and Xiaogang Zhang<p>
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>)
</p>
</div>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="cma_es.html"><img src="../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../optimization.html"><img src="../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="gd_opt/introduction.html"><img src="../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
</body>
</html>

129
doc/html/math_toolkit/gd_opt/gradient_descent.html Normal file
View File

@@ -0,0 +1,129 @@
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Gradient Desccent</title>
<link rel="stylesheet" href="../../math.css" type="text/css">
<meta name="generator" content="DocBook XSL Stylesheets Vsnapshot">
<link rel="home" href="../../index.html" title="Math Toolkit 4.2.1">
<link rel="up" href="../gd_opt.html" title="Gradient Based Optimizers">
<link rel="prev" href="introduction.html" title="Introduction">
<link rel="next" href="nesterov.html" title="Nesterov Gradient Desccent">
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
<table cellpadding="2" width="100%"><tr>
<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../boost.png"></td>
<td align="center"><a href="../../../../../../index.html">Home</a></td>
<td align="center"><a href="../../../../../../libs/libraries.htm">Libraries</a></td>
<td align="center"><a href="http://www.boost.org/users/people.html">People</a></td>
<td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td>
<td align="center"><a href="../../../../../../more/index.htm">More</a></td>
</tr></table>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="introduction.html"><img src="../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../gd_opt.html"><img src="../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="nesterov.html"><img src="../../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
<div class="section">
<div class="titlepage"><div><div><h3 class="title">
<a name="math_toolkit.gd_opt.gradient_descent"></a><a class="link" href="gradient_descent.html" title="Gradient Desccent">Gradient Desccent</a>
</h3></div></div></div>
<h5>
<a name="math_toolkit.gd_opt.gradient_descent.h0"></a>
<span class="phrase"><a name="math_toolkit.gd_opt.gradient_descent.synopsis"></a></span><a class="link" href="gradient_descent.html#math_toolkit.gd_opt.gradient_descent.synopsis">Synopsis</a>
</h5>
<pre class="programlisting"><span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">math</span><span class="special">/</span><span class="identifier">optimization</span><span class="special">/</span><span class="identifier">gradient_descent</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
<span class="keyword">template</span><span class="special">&lt;</span><span class="keyword">typename</span> <span class="identifier">ArgumentContainer</span><span class="special">,</span>
<span class="keyword">typename</span> <span class="identifier">RealType</span><span class="special">,</span>
<span class="keyword">class</span> <span class="identifier">Objective</span><span class="special">,</span>
<span class="keyword">class</span> <span class="identifier">InitializationPolicy</span><span class="special">,</span>
<span class="keyword">class</span> <span class="identifier">ObjectiveEvalPolicy</span><span class="special">,</span>
<span class="keyword">class</span> <span class="identifier">GradEvalPolicy</span><span class="special">&gt;</span>
<span class="keyword">class</span> <span class="identifier">gradient_descent</span> <span class="special">{</span>
<span class="keyword">public</span><span class="special">:</span>
<span class="keyword">void</span> <span class="identifier">step</span><span class="special">();</span>
<span class="special">}</span>
<span class="comment">/* Convenience overloads */</span>
<span class="comment">/* make gradient descent by providing
** objective function
** variables to optimize over
** optionally learing rate
*
* requires that code is written using boost::math::differentiation::rvar
*/</span>
<span class="keyword">template</span><span class="special">&lt;</span><span class="keyword">class</span> <span class="identifier">Objective</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">ArgumentContainer</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">RealType</span><span class="special">&gt;</span>
<span class="keyword">auto</span> <span class="identifier">make_gradient_descent</span><span class="special">(</span><span class="identifier">Objective</span><span class="special">&amp;&amp;</span> <span class="identifier">obj</span><span class="special">,</span> <span class="identifier">ArgumentContainer</span><span class="special">&amp;</span> <span class="identifier">x</span><span class="special">,</span> <span class="identifier">RealType</span> <span class="identifier">lr</span> <span class="special">=</span> <span class="identifier">RealType</span><span class="special">{</span> <span class="number">0.01</span> <span class="special">});</span>
<span class="comment">/* make gradient descent by providing
* objective function
** variables to optimize over
** learning rate (not optional)
** initialization policy
*
* requires that code is written using boost::math::differentiation::rvar
*/</span>
<span class="keyword">template</span><span class="special">&lt;</span><span class="keyword">class</span> <span class="identifier">Objective</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">ArgumentContainer</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">RealType</span><span class="special">,</span> <span class="keyword">class</span> <span class="identifier">InitializationPolicy</span><span class="special">&gt;</span>
<span class="keyword">auto</span> <span class="identifier">make_gradient_descent</span><span class="special">(</span><span class="identifier">Objective</span><span class="special">&amp;&amp;</span> <span class="identifier">obj</span><span class="special">,</span>
<span class="identifier">ArgumentContainer</span><span class="special">&amp;</span> <span class="identifier">x</span><span class="special">,</span>
<span class="identifier">RealType</span> <span class="identifier">lr</span><span class="special">,</span>
<span class="identifier">InitializationPolicy</span><span class="special">&amp;&amp;</span> <span class="identifier">ip</span><span class="special">);</span>
<span class="comment">/* make gradient descent by providing
** objective function
** variables to optimize over
** learning rate (not optional)
** variable initialization policy
** objective evaluation policy
** gradient evaluation policy
*
* code does not have to use boost::math::differentiation::rvar
*/</span>
<span class="keyword">template</span><span class="special">&lt;</span><span class="keyword">typename</span> <span class="identifier">ArgumentContainer</span><span class="special">,</span>
<span class="keyword">typename</span> <span class="identifier">RealType</span><span class="special">,</span>
<span class="keyword">class</span> <span class="identifier">Objective</span><span class="special">,</span>
<span class="keyword">class</span> <span class="identifier">InitializationPolicy</span><span class="special">,</span>
<span class="keyword">class</span> <span class="identifier">ObjectiveEvalPolicy</span><span class="special">,</span>
<span class="keyword">class</span> <span class="identifier">GradEvalPolicy</span><span class="special">&gt;</span>
<span class="keyword">auto</span> <span class="identifier">make_gradient_descent</span><span class="special">(</span><span class="identifier">Objective</span><span class="special">&amp;&amp;</span> <span class="identifier">obj</span><span class="special">,</span>
<span class="identifier">ArgumentContainer</span><span class="special">&amp;</span> <span class="identifier">x</span><span class="special">,</span>
<span class="identifier">RealType</span><span class="special">&amp;</span> <span class="identifier">lr</span><span class="special">,</span>
<span class="identifier">InitializationPolicy</span><span class="special">&amp;&amp;</span> <span class="identifier">ip</span><span class="special">,</span>
<span class="identifier">ObjectiveEvalPolicy</span><span class="special">&amp;&amp;</span> <span class="identifier">oep</span><span class="special">,</span>
<span class="identifier">GradEvalPolicy</span><span class="special">&amp;&amp;</span> <span class="identifier">gep</span><span class="special">)</span>
</pre>
<p>
Gradient descent iteratively updates parameters <code class="computeroutput"><span class="identifier">x</span></code>
in the direction opposite to the gradient of the objective function (minimizing
the objective).
</p>
<pre class="programlisting"><span class="identifier">x</span><span class="special">[</span><span class="identifier">i</span><span class="special">]</span> <span class="special">-=</span> <span class="identifier">lr</span> <span class="special">*</span> <span class="identifier">g</span><span class="special">[</span><span class="identifier">i</span><span class="special">]</span>
</pre>
<p>
where <code class="computeroutput"><span class="identifier">lr</span></code> is a user defined
learning rate. For a more complete decription of the theoretical principle
check <a href="https://en.wikipedia.org/wiki/Gradient_descent" target="_top">the wikipedia
page</a>
</p>
<p>
The implementation delegates: - the initialization of differentiable variables
to an initialization policy - objective evaluation to an objective evaluation
policy - the gradient computation to a gradient evaluation policy - the parameter
updates to an update policy
</p>
</div>
<div class="copyright-footer">Copyright © 2006-2021 Nikhar Agrawal, Anton Bikineev, Matthew Borland,
Paul A. Bristow, Marco Guazzone, Christopher Kormanyos, Hubert Holin, Bruno
Lalande, John Maddock, Evan Miller, Jeremy Murphy, Matthew Pulver, Johan Råde,
Gautam Sewani, Benjamin Sobotta, Nicholas Thompson, Thijs van den Berg, Daryle
Walker and Xiaogang Zhang<p>
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>)
</p>
</div>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="introduction.html"><img src="../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../gd_opt.html"><img src="../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="nesterov.html"><img src="../../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
</body>
</html>

43
doc/html/math_toolkit/gd_opt/introduction.html Normal file
View File

@@ -0,0 +1,43 @@
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Introduction</title>
<link rel="stylesheet" href="../../math.css" type="text/css">
<meta name="generator" content="DocBook XSL Stylesheets Vsnapshot">
<link rel="home" href="../../index.html" title="Math Toolkit 4.2.1">
<link rel="up" href="../gd_opt.html" title="Gradient Based Optimizers">
<link rel="prev" href="../gd_opt.html" title="Gradient Based Optimizers">
<link rel="next" href="gradient_descent.html" title="Gradient Desccent">
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
<table cellpadding="2" width="100%"><tr>
<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../boost.png"></td>
<td align="center"><a href="../../../../../../index.html">Home</a></td>
<td align="center"><a href="../../../../../../libs/libraries.htm">Libraries</a></td>
<td align="center"><a href="http://www.boost.org/users/people.html">People</a></td>
<td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td>
<td align="center"><a href="../../../../../../more/index.htm">More</a></td>
</tr></table>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="../gd_opt.html"><img src="../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../gd_opt.html"><img src="../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="gradient_descent.html"><img src="../../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
<div class="section"><div class="titlepage"><div><div><h3 class="title">
<a name="math_toolkit.gd_opt.introduction"></a><a class="link" href="introduction.html" title="Introduction">Introduction</a>
</h3></div></div></div></div>
<div class="copyright-footer">Copyright © 2006-2021 Nikhar Agrawal, Anton Bikineev, Matthew Borland,
Paul A. Bristow, Marco Guazzone, Christopher Kormanyos, Hubert Holin, Bruno
Lalande, John Maddock, Evan Miller, Jeremy Murphy, Matthew Pulver, Johan Råde,
Gautam Sewani, Benjamin Sobotta, Nicholas Thompson, Thijs van den Berg, Daryle
Walker and Xiaogang Zhang<p>
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>)
</p>
</div>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="../gd_opt.html"><img src="../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../gd_opt.html"><img src="../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="gradient_descent.html"><img src="../../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
</body>
</html>

43
doc/html/math_toolkit/gd_opt/lbfgs.html Normal file
View File

@@ -0,0 +1,43 @@
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>L-BFGS</title>
<link rel="stylesheet" href="../../math.css" type="text/css">
<meta name="generator" content="DocBook XSL Stylesheets Vsnapshot">
<link rel="home" href="../../index.html" title="Math Toolkit 4.2.1">
<link rel="up" href="../gd_opt.html" title="Gradient Based Optimizers">
<link rel="prev" href="nesterov.html" title="Nesterov Gradient Desccent">
<link rel="next" href="../../poly.html" title="Chapter 12. Polynomials and Rational Functions">
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
<table cellpadding="2" width="100%"><tr>
<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../boost.png"></td>
<td align="center"><a href="../../../../../../index.html">Home</a></td>
<td align="center"><a href="../../../../../../libs/libraries.htm">Libraries</a></td>
<td align="center"><a href="http://www.boost.org/users/people.html">People</a></td>
<td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td>
<td align="center"><a href="../../../../../../more/index.htm">More</a></td>
</tr></table>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="nesterov.html"><img src="../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../gd_opt.html"><img src="../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="../../poly.html"><img src="../../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
<div class="section"><div class="titlepage"><div><div><h3 class="title">
<a name="math_toolkit.gd_opt.lbfgs"></a><a class="link" href="lbfgs.html" title="L-BFGS">L-BFGS</a>
</h3></div></div></div></div>
<div class="copyright-footer">Copyright © 2006-2021 Nikhar Agrawal, Anton Bikineev, Matthew Borland,
Paul A. Bristow, Marco Guazzone, Christopher Kormanyos, Hubert Holin, Bruno
Lalande, John Maddock, Evan Miller, Jeremy Murphy, Matthew Pulver, Johan Råde,
Gautam Sewani, Benjamin Sobotta, Nicholas Thompson, Thijs van den Berg, Daryle
Walker and Xiaogang Zhang<p>
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>)
</p>
</div>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="nesterov.html"><img src="../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../gd_opt.html"><img src="../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="../../poly.html"><img src="../../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
</body>
</html>

43
doc/html/math_toolkit/gd_opt/nesterov.html Normal file
View File

@@ -0,0 +1,43 @@
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Nesterov Gradient Desccent</title>
<link rel="stylesheet" href="../../math.css" type="text/css">
<meta name="generator" content="DocBook XSL Stylesheets Vsnapshot">
<link rel="home" href="../../index.html" title="Math Toolkit 4.2.1">
<link rel="up" href="../gd_opt.html" title="Gradient Based Optimizers">
<link rel="prev" href="gradient_descent.html" title="Gradient Desccent">
<link rel="next" href="lbfgs.html" title="L-BFGS">
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
<table cellpadding="2" width="100%"><tr>
<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../boost.png"></td>
<td align="center"><a href="../../../../../../index.html">Home</a></td>
<td align="center"><a href="../../../../../../libs/libraries.htm">Libraries</a></td>
<td align="center"><a href="http://www.boost.org/users/people.html">People</a></td>
<td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td>
<td align="center"><a href="../../../../../../more/index.htm">More</a></td>
</tr></table>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="gradient_descent.html"><img src="../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../gd_opt.html"><img src="../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="lbfgs.html"><img src="../../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
<div class="section"><div class="titlepage"><div><div><h3 class="title">
<a name="math_toolkit.gd_opt.nesterov"></a><a class="link" href="nesterov.html" title="Nesterov Gradient Desccent">Nesterov Gradient Desccent</a>
</h3></div></div></div></div>
<div class="copyright-footer">Copyright © 2006-2021 Nikhar Agrawal, Anton Bikineev, Matthew Borland,
Paul A. Bristow, Marco Guazzone, Christopher Kormanyos, Hubert Holin, Bruno
Lalande, John Maddock, Evan Miller, Jeremy Murphy, Matthew Pulver, Johan Råde,
Gautam Sewani, Benjamin Sobotta, Nicholas Thompson, Thijs van den Berg, Daryle
Walker and Xiaogang Zhang<p>
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>)
</p>
</div>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="gradient_descent.html"><img src="../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../gd_opt.html"><img src="../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="lbfgs.html"><img src="../../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
</body>
</html>

1
doc/math.qbk
View File

@@ -739,6 +739,7 @@ and as a CD ISBN 0-9504833-2-X 978-0-9504833-2-0, Classification 519.2-dc22.
[include optimization/jso.qbk]
[include optimization/random_search.qbk]
[include optimization/cma_es.qbk]
[include optimization/gradient_optimizers.qbk]
[endmathpart] [/mathpart optimization Optimization]
[mathpart poly Polynomials and Rational Functions]

101
doc/optimization/gradient_optimizers.qbk Normal file
View File

@@ -0,0 +1,101 @@
[/
Copyright (c) 2025-2026 Maksym Zhelyeznyakov
Use, modification and distribution are subject to the
Boost Software License, Version 1.0. (See accompanying file
LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
]
[section:gd_opt Gradient Based Optimizers]
Gradient based optimizers are algorithms that use the gradient of a funciton to iteratively find locally extreme points of functions over a set of parameters. This sections provides a description of a set of gradient optimizers. The optimizers are written with `boost::math::differentiation::reverse_mode::rvar` in mind, however if a way to evaluate the funciton and its gradient is provided, the optimizers should work in exactly the same way.
[section:introduction Introduction]
[endsect] [/section:introduction]
[section:gradient_descent Gradient Desccent]
[heading Synopsis]
``
#include <boost/math/optimization/gradient_descent.hpp>
template<typename ArgumentContainer,
typename RealType,
class Objective,
class InitializationPolicy,
class ObjectiveEvalPolicy,
class GradEvalPolicy>
class gradient_descent {
public:
void step();
}
/* Convenience overloads */
/* make gradient descent by providing
** objective function
** variables to optimize over
** optionally learing rate
*
* requires that code is written using boost::math::differentiation::rvar
*/
template<class Objective, typename ArgumentContainer, typename RealType>
auto make_gradient_descent(Objective&& obj, ArgumentContainer& x, RealType lr = RealType{ 0.01 });
/* make gradient descent by providing
* objective function
** variables to optimize over
** learning rate (not optional)
** initialization policy
*
* requires that code is written using boost::math::differentiation::rvar
*/
template<class Objective, typename ArgumentContainer, typename RealType, class InitializationPolicy>
auto make_gradient_descent(Objective&& obj,
ArgumentContainer& x,
RealType lr,
InitializationPolicy&& ip);
/* make gradient descent by providing
** objective function
** variables to optimize over
** learning rate (not optional)
** variable initialization policy
** objective evaluation policy
** gradient evaluation policy
*
* code does not have to use boost::math::differentiation::rvar
*/
template<typename ArgumentContainer,
typename RealType,
class Objective,
class InitializationPolicy,
class ObjectiveEvalPolicy,
class GradEvalPolicy>
auto make_gradient_descent(Objective&& obj,
ArgumentContainer& x,
RealType& lr,
InitializationPolicy&& ip,
ObjectiveEvalPolicy&& oep,
GradEvalPolicy&& gep)
``
Gradient descent iteratively updates parameters `x` in the direction opposite to the gradient of the objective function (minimizing the objective).
``
x[i] -= lr * g[i]
``
where `lr` is a user defined learning rate. For a more complete decription of the theoretical principle check [@https://en.wikipedia.org/wiki/Gradient_descent the wikipedia page]
The implementation delegates:
- the initialization of differentiable variables to an initialization policy
- objective evaluation to an objective evaluation policy
- the gradient computation to a gradient evaluation policy
- the parameter updates to an update policy
[endsect] [/section:gradient_descent]
[section:nesterov Nesterov Gradient Desccent]
[endsect] [/section:nesterov]
[section:lbfgs L-BFGS]
[endsect] [/section:lbfgs]
[endsect] [/section:gd_opt]