BN层（Pytorch讲解）

批量归一化（BN：Batch Normalization
解决在训练过程中，中间层数据分布过度异常问题，让数据分布符合正态分布
主要达到三个目的：
1.加快网络的训练和收敛的速度
2.控制梯度爆炸
3.防止梯度消失
4.防止过拟合

归一化公式：减去均值，再除以标准差

from torch import nn
import torch

2D的BN层

rgb = torch.randn(1, 3, 2, 2) # (batchsize,channel,w,h)
print(rgb)
print(rgb.shape)

conv = nn.Conv2d(3, 2, 1)
x = conv(rgb)
print(x)
print(x.shape)

bn = nn.BatchNorm2d(2)
res = bn(x)
print(res)
print(res.shape)

mean = x.mean(dim=0, keepdim=True).mean(dim=2, keepdim=True).mean(dim=3, keepdim=True)
var = ((x - mean) ** 2).mean(dim=0, keepdim=True).mean(dim=2, keepdim=True).mean(dim=3, keepdim=True)
x_hat = (x - mean) / torch.sqrt(var + 0.00001)
print(x_hat)

1D的BN层

m = nn.BatchNorm1d(2,affine=False)
input = torch.randn(2, 2)
output = m(input)

print(input)

print(input.shape)

print(output)

在这里插入代码片

mean = input.mean(dim=0)
var = ((input - mean) ** 2).mean(dim=0)

test = (input-mean)/ torch.sqrt(var + 1e-5)

test

实例：

m = nn.Linear(20, 30)
input = torch.randn(128, 20)
output = m(input)
print(output.size())

bn_layer = nn.BatchNorm1d(30,affine=False)
o2 = bn_layer(output)
o2.shape