自定义层
========
深度学习成功背后的一个因素是神经网络的灵活性：
我们可以用创造性的方式组合不同的层，从而设计出适用于各种任务的架构。
例如，研究人员发明了专门用于处理图像、文本、序列数据和执行动态规划的层。
有时我们会遇到或要自己发明一个现在在深度学习框架中还不存在的层。
在这些情况下，必须构建自定义层。本节将展示如何构建自定义层。
不带参数的层
------------
首先，我们构造一个没有任何参数的自定义层。 回忆一下在
:numref:`sec_model_construction`\ 对块的介绍， 这应该看起来很眼熟。
下面的\ ``CenteredLayer``\ 类要从其输入中减去均值。
要构建它，我们只需继承基础层类并实现前向传播功能。
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
from mxnet import np, npx
from mxnet.gluon import nn
npx.set_np()
class CenteredLayer(nn.Block):
def __init__(self, **kwargs):
super().__init__(**kwargs)
def forward(self, X):
return X - X.mean()
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
import torch
import torch.nn.functional as F
from torch import nn
class CenteredLayer(nn.Module):
def __init__(self):
super().__init__()
def forward(self, X):
return X - X.mean()
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
import tensorflow as tf
class CenteredLayer(tf.keras.Model):
def __init__(self):
super().__init__()
def call(self, inputs):
return inputs - tf.reduce_mean(inputs)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
import warnings
warnings.filterwarnings(action='ignore')
import paddle
import paddle.nn.functional as F
from paddle import nn
class CenteredLayer(nn.Layer):
def __init__(self):
super().__init__()
def forward(self, X):
return X - X.mean()
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
layer = CenteredLayer()
layer(np.array([1, 2, 3, 4, 5]))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
[07:15:32] ../src/storage/storage.cc:196: Using Pooled (Naive) StorageManager for CPU
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([-2., -1., 0., 1., 2.])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
layer = CenteredLayer()
layer(torch.FloatTensor([1, 2, 3, 4, 5]))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([-2., -1., 0., 1., 2.])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
layer = CenteredLayer()
layer(tf.constant([1, 2, 3, 4, 5]))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
layer = CenteredLayer()
layer(paddle.to_tensor([1, 2, 3, 4, 5], dtype='float32'))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[5], dtype=float32, place=Place(cpu), stop_gradient=True,
[-2., -1., 0., 1., 2.])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net = nn.Sequential()
net.add(nn.Dense(128), CenteredLayer())
net.initialize()
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net = nn.Sequential(nn.Linear(8, 128), CenteredLayer())
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net = tf.keras.Sequential([tf.keras.layers.Dense(128), CenteredLayer()])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net = nn.Sequential(nn.Linear(8, 128), CenteredLayer())
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
Y = net(np.random.uniform(size=(4, 8)))
Y.mean()
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array(3.783498e-10)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
Y = net(torch.rand(4, 8))
Y.mean()
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor(7.4506e-09, grad_fn=)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
Y = net(tf.random.uniform((4, 8)))
tf.reduce_mean(Y)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
Y = net(paddle.rand([4, 8]))
Y.mean()
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[1], dtype=float32, place=Place(cpu), stop_gradient=False,
[-0.00000000])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
class MyDense(nn.Block):
def __init__(self, units, in_units, **kwargs):
super().__init__(**kwargs)
self.weight = self.params.get('weight', shape=(in_units, units))
self.bias = self.params.get('bias', shape=(units,))
def forward(self, x):
linear = np.dot(x, self.weight.data(ctx=x.ctx)) + self.bias.data(
ctx=x.ctx)
return npx.relu(linear)
接下来，我们实例化\ ``MyDense``\ 类并访问其模型参数。
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
dense = MyDense(units=3, in_units=5)
dense.params
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
mydense0_ (
Parameter mydense0_weight (shape=(5, 3), dtype=)
Parameter mydense0_bias (shape=(3,), dtype=)
)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
class MyLinear(nn.Module):
def __init__(self, in_units, units):
super().__init__()
self.weight = nn.Parameter(torch.randn(in_units, units))
self.bias = nn.Parameter(torch.randn(units,))
def forward(self, X):
linear = torch.matmul(X, self.weight.data) + self.bias.data
return F.relu(linear)
接下来，我们实例化\ ``MyLinear``\ 类并访问其模型参数。
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
linear = MyLinear(5, 3)
linear.weight
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Parameter containing:
tensor([[ 0.1775, -1.4539, 0.3972],
[-0.1339, 0.5273, 1.3041],
[-0.3327, -0.2337, -0.6334],
[ 1.2076, -0.3937, 0.6851],
[-0.4716, 0.0894, -0.9195]], requires_grad=True)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
class MyDense(tf.keras.Model):
def __init__(self, units):
super().__init__()
self.units = units
def build(self, X_shape):
self.weight = self.add_weight(name='weight',
shape=[X_shape[-1], self.units],
initializer=tf.random_normal_initializer())
self.bias = self.add_weight(
name='bias', shape=[self.units],
initializer=tf.zeros_initializer())
def call(self, X):
linear = tf.matmul(X, self.weight) + self.bias
return tf.nn.relu(linear)
接下来，我们实例化\ ``MyDense``\ 类并访问其模型参数。
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
dense = MyDense(3)
dense(tf.random.uniform((2, 5)))
dense.get_weights()
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
[array([[-0.013614 , -0.01669732, 0.02921283],
[ 0.03179312, 0.0889833 , -0.02140525],
[ 0.05018818, 0.02113006, 0.07468227],
[ 0.03596197, -0.0285063 , 0.04013855],
[-0.0061096 , -0.00112533, 0.01261374]], dtype=float32),
array([0., 0., 0.], dtype=float32)]
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
class MyLinear(nn.Layer):
def __init__(self, in_units, units):
super().__init__()
self.weight = paddle.create_parameter(shape=(in_units, units), dtype='float32')
self.bias = paddle.create_parameter(shape=(units,), dtype='float32')
def forward(self, X):
linear = paddle.matmul(X, self.weight) + self.bias
return F.relu(linear)
接下来，我们实例化\ ``MyLinear``\ 类并访问其模型参数。
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
linear = MyLinear(5, 3)
linear.weight
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Parameter containing:
Tensor(shape=[5, 3], dtype=float32, place=Place(cpu), stop_gradient=False,
[[ 0.27573973, 0.74489242, 0.31523722],
[ 0.12920702, 0.38870257, 0.47107095],
[-0.19269449, -0.55530524, 0.79270011],
[-0.58781934, -0.50390607, -0.54891467],
[ 0.06385410, -0.79282713, -0.10914403]])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
dense.initialize()
dense(np.random.uniform(size=(2, 5)))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([[0. , 0.01633355, 0. ],
[0. , 0.01581812, 0. ]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
linear(torch.rand(2, 5))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([[0., 0., 0.],
[0., 0., 0.]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
dense(tf.random.uniform((2, 5)))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
linear(paddle.randn([2, 5]))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[2, 3], dtype=float32, place=Place(cpu), stop_gradient=False,
[[0. , 0. , 0.77909648],
[0. , 0.65021682, 1.60768795]])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net = nn.Sequential()
net.add(MyDense(8, in_units=64),
MyDense(1, in_units=8))
net.initialize()
net(np.random.uniform(size=(2, 64)))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([[0.06508517],
[0.0615553 ]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net = nn.Sequential(MyLinear(64, 8), MyLinear(8, 1))
net(torch.rand(2, 64))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([[0.],
[0.]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net = tf.keras.models.Sequential([MyDense(8), MyDense(1)])
net(tf.random.uniform((2, 64)))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net = nn.Sequential(MyLinear(64, 8), MyLinear(8, 1))
net(paddle.rand([2, 64]))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[2, 1], dtype=float32, place=Place(cpu), stop_gradient=False,
[[1.46890318],
[1.54701924]])
.. raw:: html
.. raw:: html
.. raw:: html
`Discussions `__
.. raw:: html
.. raw:: html
`Discussions `__
.. raw:: html
.. raw:: html
`Discussions `__
.. raw:: html
.. raw:: html
`Discussions `__
.. raw:: html
.. raw:: html