【点云】Pointet的读后感(CVPR2016):3.分类网络部分

先贴上pointnet原文当中的网络结构,后面再慢慢分析.可以看到图中有两个网络.一个是分类网络,另一个是语义分割网络.先说分类网络.可以看到分割网络只是在最后把全局信息重复了n次然后concat到nx64那层网络之后…呀,顺序有点问题,LZ连分类网络还没有解释呢?好了,从分类网络说起.我们可以看到输入是nx3,n代表的是输入点云的数量,一般取128/256/512/1024/2048/4096,当然取得点越多,获得的信息越多,accuracy相对也会越高,感觉有种在玩2048的感觉,O(∩_∩)O哈哈~3就是(x, y, z)的三维点的坐标.后面接着一个input transformation,这个是部分是嵌入了一个T-Net,T-Net学习的是一个3x3的矩阵,是为了解决输入特征的旋转不变性.相当于是个R,但是这里LZ还是存在一个疑惑,就是这个并没有在loss中进行归一化,怎么保证乘的3x3矩阵满足旋转矩阵的几个性质(⊙o⊙)?下面贴出T-Net的代码:definput_transform_net(point_cloud,is_training,bn_decayNone,K3): Input (XYZ) Transform Net, input is BxNx3 gray image Return: Transformation matrix of size 3xK batch_sizepoint_cloud.get_shape()[0].value num_pointpoint_cloud.get_shape()[1].value# 从尾巴增加一维向量input_imagetf.expand_dims(point_cloud,-1)# print(input_image.shape)# 定义了一个卷积核input_image[B, N, 3, 1], number_output_channels 64,kernel_size [1, 3]#nettf_util.conv2d(input_image,64,[1,3],paddingVALID,stride[1,1],bnTrue,is_trainingis_training,scopetconv1,bn_decaybn_decay)# print(tconv1 , net.shape)nettf_util.conv2d(net,128,[1,1],paddingVALID,stride[1,1],bnTrue,is_trainingis_training,scopetconv2,bn_decaybn_decay)# print(tconv2,net.shape)nettf_util.conv2d(net,1024,[1,1],paddingVALID,stride[1,1],bnTrue,is_trainingis_training,scopetconv3,bn_decaybn_decay)# print(tconv3,net.shape)nettf_util.max_pool2d(net,[num_point,1],paddingVALID,scopetmaxpool)# print(tmaxpool,net.shape)nettf.reshape(net,[batch_size,-1])# print(net.shape)nettf_util.fully_connected(net,512,bnTrue,is_trainingis_training,scopetfc1,bn_decaybn_decay)# print(tfc1,net.shape)nettf_util.fully_connected(net,256,bnTrue,is_trainingis_training,scopetfc2,bn_decaybn_decay)# print(tfc2,net.shape)withtf.variable_scope(transform_XYZ)assc:assert(K3)weightstf.get_variable(weights,[256,3*K],initializertf.constant_initializer(0.0),dtypetf.float32)biasestf.get_variable(biases,[3*K],initializertf.constant_initializer(0.0),dtypetf.float32)biasestf.constant([1,0,0,0,1,0,0,0,1],dtypetf.float32)transformtf.matmul(net,weights)transformtf.nn.bias_add(transform,biases)transformtf.reshape(transform,[batch_size,3,K])# print(transform.shape)returntransform可以看到T-Net其实也就是一个小型网络,可以看到3个卷积层,两个全连接层,最后得到的[batch_size, 256]维的tensor乘以[256, 3x3], 得到[batch_size, 9]后,加上biases,再通过reshape得到[batch_size, 3, 3]的tensor. LZ一开始没弄明白到底T-Net是怎么进行训练的, 后面可以理解了,把训练得到的tensor是直接乘到原始的数据上去的,可以直接看做是一个网络的嵌入,这样在优化loss的时候,就可以直接对T-Net也同时进行优化.同理,对于feature transform也是同样的,对高维也就是64维特征空间上进行旋转,所以训练出来的tensor的维度是[batch_size, 64, 64], 这里先贴出feature transform的代码,可以看到和input transform非常相似deffeature_transform_net(inputs,is_training,bn_decayNone,K64): Feature Transform Net, input is BxNx1xK Return: Transformation matrix of size KxK batch_sizeinputs.get_shape()[0].value num_pointinputs.get_shape()[1].value nettf_util.conv2d(inputs,64,[1,1],paddingVALID,stride[1,1],bnTrue,is_trainingis_training,scopetconv1,bn_decaybn_decay)# print(tconv1 , net.shape)nettf_util.conv2d(net,128,[1,1],paddingVALID,stride[1,1],bnTrue,is_trainingis_training,scopetconv2,bn_decaybn_decay)# print(tconv2 , net.shape)nettf_util.conv2d(net,1024,[1,1],paddingVALID,stride[1,1],bnTrue,is_trainingis_training,scopetconv3,bn_decaybn_decay)# print(tconv3 , net.shape)nettf_util.max_pool2d(net,[num_point,1],paddingVALID,scopetmaxpool)# print(tmaxpool , net.shape)nettf.reshape(net,[batch_size,-1])# print(reshape, net.shape)nettf_util.fully_connected(net,512,bnTrue,is_trainingis_training,scopetfc1,bn_decaybn_decay)# print(tfc1 , net.shape)nettf_util.fully_connected(net,256,bnTrue,is_trainingis_training,scopetfc2,bn_decaybn_decay)# print(tfc2 , net.shape)withtf.variable_scope(transform_feat)assc:weightstf.get_variable(weights,[256,K*K],initializertf.constant_initializer(0.0),dtypetf.float32)biasestf.get_variable(biases,[K*K],initializertf.constant_initializer(0.0),dtypetf.float32)biasestf.constant(np.eye(K).flatten(),dtypetf.float32)transformtf.matmul(net,weights)transformtf.nn.bias_add(transform,biases)# print(transform , transform.shape)transformtf.reshape(transform,[batch_size,K,K])# print(transform_reshape , transform.shape)returntransform当中的多层感知机也就是图中的mlp其实就是我们通常理解的卷积层,直接贴上代码,简单明了:defget_model(point_cloud,is_training,bn_decayNone): Classification PointNet, input is BxNx3, output Bx40 batch_sizepoint_cloud.get_shape()[0].value num_pointpoint_cloud.get_shape()[1].value end_points{}withtf.variable_scope(transform_net1)assc:transforminput_transform_net(point_cloud,is_training,bn_decay,K3)point_cloud_transformedtf.matmul(point_cloud,transform)# print(point_cloud_transformed.shape)input_imagetf.expand_dims(point_cloud_transformed,-1)# net tf_util.conv2d(input_image, 64, [1,3],# paddingVALID, stride[1,1],# bnTrue, is_trainingis_training,# scopeconv1, bn_decaybn_decay)nettf_util.conv2d(input_image,64,[1,3],paddingVALID,stride[1,1],bnTrue,is_trainingis_training,scopeconv1,bn_decaybn_decay)# print(conv1 , net.shape)nettf_util.conv2d(net,64,[1,1],paddingVALID,stride[1,1],bnTrue,is_trainingis_training,scopeconv2,bn_decaybn_decay)# print(conv2 , net.shape)withtf.variable_scope(transform_net2)assc:transformfeature_transform_net(net,is_training,bn_decay,K64)end_points[transform]transform net_squeezetf.squeeze(net,axis[2])# print(net_squeeze shape , net_squeeze.shape)net_transformedtf.matmul(tf.squeeze(net,axis[2]),transform)# print(net_transformed 1:, net_transformed.shape)net_transformedtf.expand_dims(net_transformed,[2])# print(net_transformed 2:, net_transformed.shape)nettf_util.conv2d(net_transformed,64,[1,1],paddingVALID,stride[1,1],bnTrue,is_trainingis_training,scopeconv3,bn_decaybn_decay)# print(conv3 , net.shape)nettf_util.conv2d(net,128,[1,1],paddingVALID,stride[1,1],bnTrue,is_trainingis_training,scopeconv4,bn_decaybn_decay)# print(conv4 , net.shape)nettf_util.conv2d(net,1024,[1,1],paddingVALID,stride[1,1],bnTrue,is_trainingis_training,scopeconv5,bn_decaybn_decay)# print(conv5 , net.shape)# Symmetric function: max poolingnettf_util.max_pool2d(net,[num_point,1],paddingVALID,scopemaxpool)# print(max_pool , net.shape)nettf.reshape(net,[batch_size,-1])# print(reshape , net.shape)nettf_util.fully_connected(net,512,bnTrue,is_trainingis_training,scopefc1,bn_decaybn_decay)# print(fc1 , net.shape)nettf_util.dropout(net,keep_prob0.7,is_trainingis_training,scopedp1)nettf_util.fully_connected(net,256,bnTrue,is_trainingis_training,scopefc2,bn_decaybn_decay)# print(fc2 , net.shape)nettf_util.dropout(net,keep_prob0.7,is_trainingis_training,scopedp2)nettf_util.fully_connected(net,40,activation_fnNone,scopefc3)# print(fc3 , net.shape)returnnet,end_points可以看到一共有五个卷积层,主要的作用就是就是把原始数据三维的特征,也就是xyz三个坐标的值升维到1024维特征,也就是最后设置的通道数,这怎么理解呢?如果直接用max pooling操作,相当于我们只取了一个点,但是一个点怎么能够代替所有点的特征呢?那么就需要对每个点的特征进行升维操作,知道最后max pooling之后的一个点的特征可以概括出所有点的特征,就可以了,实验证明这个操作是会丢失信息的,虽然解释上可以这么进行理解,但是实际上这个操作还是会丢失很多其他点和全局的信息,虽然结果已经比voxnet好很多,但是也有提升的空间在.最后用全连接层进行连接,共有3个全连接层,使用dropout来防止网络过拟合,但实际上网络还是过拟合了/(ㄒoㄒ)/~因为使用modelnet40,所以,最后的全连接的数量是40,这个都是基础了,如果有不理解的部分建议看LZ的cs231的那部分,估计两天就差不多了解个大概了.我们看下loss的具体设置defget_loss(pred,label,end_points,reg_weight0.001): pred: B*NUM_CLASSES, label: B, losstf.nn.sparse_softmax_cross_entropy_with_logits(logitspred,labelslabel)classify_losstf.reduce_mean(loss)tf.summary.scalar(classify loss,classify_loss)# Enforce the transformation as orthogonal matrixtransformend_points[transform]# BxKxKKtransform.get_shape()[1].value# transform_trans tf.transpose(transform, perm[0, 2, 1])# print(transform_trans.shape, transform_trans)mat_difftf.matmul(transform,tf.transpose(transform,perm[0,2,1]))# print(mat_diff: , mat_diff)mat_diff-tf.constant(np.eye(K),dtypetf.float32)mat_diff_losstf.nn.l2_loss(mat_diff)tf.summary.scalar(mat loss,mat_diff_loss)returnclassify_lossmat_diff_loss*reg_weight这部分也超级好理解,通常对于分类问题,我们直接使用的就是softmax loss,代码中也体现出来了,这里加了一个正则项,一则是为了防止过拟合,二则也是为了满足旋转矩阵的性质,至于为什么三维的input transform不用,待考证,可能效果不是很好???这个就是分类网络的部分,最后评价标准:pred_valnp.argmax(pred_val,1)correctnp.sum(pred_valcurrent_label[start_idx:end_idx])total_correctcorrect total_seenBATCH_SIZE loss_sum(loss_val*BATCH_SIZE)foriinrange(start_idx,end_idx):lcurrent_label[i]total_seen_class[l]1total_correct_class[l](pred_val[i-start_idx]l)log_string(eval mean loss: %f%(loss_sum/float(total_seen)))log_string(eval accuracy: %f%(total_correct/float(total_seen)))log_string(eval avg class acc: %f%(np.mean(np.array(total_correct_class)/np.array(total_seen_class,dtypenp.float))))eval_mean_lossloss_sum/float(total_seen)eval_accuracytotal_correct/float(total_seen)eval_avg_class_accnp.mean(np.array(total_correct_class)/np.array(total_seen_class,dtypenp.float))很好如果predict是1, label也是1,就是计数加1,总共计数除以总数得到accuracy就是我们需要的结果,还有一个问题实际在paper中得到的训练结果有89.2%,LZ怎么尝试都只有大概88.5%左右,(⊙v⊙)嗯,可能Local optimum.