Neural nets with superlinear vcdimension springerlink. The vapnikchervonenkis dimension of graph and recursive. Vapnikchervonenkis dimension of neural nets 4 less obvious to see is that there exists no set d0 r2 consisting of 4 or more points which is shattered by ft2, i. Radial basis function neural networks have superlinear vc. This paper shows that neural networks which use continuous acti vation functions have vc dimension at least as large as the square of the number of weights w. We give upper bounds on the vapnikchervonenkis dimension and pseudodimen sion of twolayer neural networks that use the standard. Then there are 32 5by5 filters with stride 2 in height and width. Our construction was originally motivated by a related one, given in 7, which showed that realnumber programs in the blum shub smale model of computation 4 with running time t have vc. There exist neural networks n consisting of linear threshold gates whose vcdimension scales proportional to w logw, where w is the number of parameters of n. Abstract we prove new upper and lower bounds on the vcdimension of deep neural networks with the relu activation function.
Binary weights are the extreme case of discrete couplings with obvious advantages in biological and technical implementations. Vc dimension of multilayer neural networks mathematics. Structural analysis and optimization of convolutional. Pdf vapnikchervonenkis dimension of recurrent neural. Why dont vc dimension and generalization bounds apply to. Polynomial bounds for vc dimension of sigmoidal and general. The chapter establishes various elementary results, and discusses how to estimate the vc dimension in several examples of interest in neural network theory. Vcdimension, pseudodimension, neural networks, relu activation function, statistical learning theory 1. Pdf vapnikchervonenkis dimension of recurrent neural networks. S 0,1 can be computed by n with some assignment of real numbers to its weights. Since graphs are a more general form than sequences, the question arises how this will affect the vc dimension of gnns and recnns.
It turns out, however, that in this case the typical and the extreme behavior of the network can be rather different. Nearlytight vcdimension and pseudodimension bounds for piecewise linear neural networks authors. Nearlytight vcdimension bounds for piecewise linear neural. This is generally to try to show that certain classes of circuits cant compute certain functions, or more commonly, to give learning bounds for neural networks. For an ordinary neural network, the vc dimension is roughly equal to the number of weights. The nearlytight bound of the vc dimension for any multilayer neural network, with a piecewise linear function, was used as calculated in the paper. Vc dimension of neural networks 3 in the above examples, one obtains, respectively.
Observe that, in the third example, sets of three points that are in a straight line cannot be shattered. Why is deep learning hyped despite bad vc dimension. We establish superlinear lower bounds on the vapnikchervonenkis vc dimension of neural networks with one hidden layer and local receptive field neurons. It was shown previously that the ability of recurrent neural networks to process sequences increases the vc dim when compared to the vc dim of neural networks, which are limited to processing vectors. Bartlett, nick harvey, chris liaw, abbas mehrabian submitted on 8 mar 2017 v1, last revised 16 oct 2017 this version, v3. Vcdimension, pseudodimension, neural networks, relu activation function, statistical learning theory. Vapnikchervonenkis dimension of neural nets citeseerx. For instance, vcdimension can be used for different types on networks or even things other than neural networks while number of neurons is only useful between very specific models where the activation function, signals basic sums vs. These successes motivate a renewed study of these networks theoretical. For any assignment of values to its internal parameters.
Pdf most of the work on the vapnikchervonenkis dimension of neural networks has been focused on feedforward networks. This follows immediately from the following theorem. Vc dimension, neural networks, relu activation function extended abstract. Electronic proceedings of the 2017 conference on learning theory. Such systems learn to perform tasks by considering examples, generally without being programmed with taskspecific rules. Letting w be the number of weights and l be the number of layers, we prove that the vcdimension is o wllog w and. In the figure below the input is 28by28 matrix with 1 channel. The vcdim has been widely studied in neural networks. Almost linear vc dimension bounds for piecewise polynomial networks 191 are in fact tight up to a constant. Department of mathematics, rutgers, the state university of new jersey. You want to find the largest size of a data set suc. The vapnikchervonenkis dimension vcdimensionn of a neural net n with n input nodes is defined as the size of the largest set s.
Adversarial vcdimension and sample complexity of neural networks. Neural networks with quadratic vc dimension nips proceedings. Nearlytight vc dimension and pseudodimension bounds for piecewise linear neural networks authors. Nearlytight vcdimension and pseudodimension bounds for piecewise linear neural networks. The information capacity of a perceptron is intensively discussed in sir david mackays book which summarizes work by thomas cover. However, the proof used in ks97 to establish the lower bound made use of the fact that the number of layers can grow with w. Polynomial bounds for vc dimension of sigmoidal and. In this section, we present a useful result relating linear vectorspace dimension to the vcdimension. Introduction deep neural networks underlie many of the recent breakthroughs in applied machine learning, particularly in image and speech recognition. Pdf vc dimension of neural networks semantic scholar. Nonetheless, the vc dimension is 3 because some 3element set can be shattered. Letting w be the number of weights and lbe the number of layers, we prove that the vcdimension is owllogw, and provide examples with vcdimension wllogwl. Number of training points that can be classified exactly is vc dimension.
This suggests that the number of training examples should grow roughly. We prove new upper and lower bounds on the vc dimension of deep neural networks with the relu activation function. The vapnikchervonenkis dimension vcdim is a measure of model complexity that allows to bound the difference between the sum of the squared errors made on the training dataset and the sum of the squared errors made on the testing dataset on the basis of the number of training examples vapnik, 1998. Vapnikchervonenkis dimension of neural networks with. For instance, vc dimension can be used for different types on networks or even things other than neural networks while number of neurons is only useful between very specific models where the activation function, signals basic sums vs. A capacity scaling law for artificial neural networks. The full version with all the proofs appears as arxiv.
R n which is shattered by n in the sense that every function f. The vcdimension and pseudodimension of twolayer neural. Pdf neural networks with quadratic vc dimension eduardo d. The vc dimension of a neural network is bounded as follows 234235. Artificial neural networks ann or connectionist systems are computing systems vaguely inspired by the biological neural networks that constitute animal brains. My guess is that the vc dimension actually grows exponentially as the number of layers l increases. While the vcdimension of networks with piecewise polynomial activation functions is well understood, most applications of neural networks use the logistic sigmoid function, or gaussian radial basis function. Vc dimension of multilayer neural networks, range queries. For a typical neural network, that would mean iterating over the data set until it make no more errors. The paper establishes various elementary results, and discusses how to estimate the vc dimension in several examples of interest in neural network theory. Adversarial vcdimension and sample complexity of neural. Bartlett, nick harvey, christopher liaw, abbas mehrabian.
Vc dimension of multilayer neural networks, range queries instructors. This is not quite possible, because the products between the wis which are inputs in this context and the heavisides are not allowed. Vcdimension of neural networks let c be a concept class over rr with vcdimension d. This chapter presents a brief introduction to vapnikchervonenkis vc dimension, a quantity which characterizes the difficulty of distributionindependent learning. We introduce a new method for proving explicit upper bounds on the vc dimension of general functional basis networks and prove as an application, for the first time, that the vc dimension of analog neural networks with the sigmoidal activation function. In general, the vc dimension of a finite classification model, which can return at most different classifiers, is at most this is an upper bound on the vc dimension. Nearlytight vcdimension bounds for piecewise linear. Unfortunately, it is not possible to compute such functions using a finite number of the arithmetic operations listed in theorem 5. Pdf neural networks with quadratic vc dimension eduardo.
Neural networks a robust approach for approximating realvalued, discretevalued or vector valued functions among the most effective general purposesupervised learning methods currently known especially for complex and hard to interpret data such as realworld sensory data the backpropagation algorithm for neural networks has. Almost linear vc dimension bounds for piecewise polynomial. Letting w be the number of weights and lbe the number of layers, we prove that the vc dimension is owllogw, and provide examples with vc dimension wllogwl. We will find this useful when it comes to bounding the vcdimension of certain neural networks. Comp9444 11s2 vcdimension and paclearning 28 vcdimensions of neural networks any function class h that can be computed by a program that takes a real input vectorx and k real parameters and involves no more than t of the following operations. I am trying to understand how the dimensions in convolutional neural network behave. This superlinear growth of the vcdimension occurs already for feedforward neural nets with two hidden layers in the case ofdiscrete network inputs. However, a tight characterization of how depth affects vcdimension was unknown prior to this work.
Vapnikchervonenkis dimension of neural nets tu graz. Vapnikchervonenkis vc dimension that avoids struc tured data and therefore provides an upper. This paper presents a brief introduction to vapnikchervonenkis vc dimension, a quantity which characterizes the difficulty of. Its vcdimension is 0 since it cannot shatter even a single point. As the main result we show that every reasonably sized standard network of radial basis function rbf neurons has vc dimension. Vcdimensions of neural networks discrete mathematics. While the vc dimension of networks with piecewise polynomial activation functions is well understood, most applications of neural networks use the logistic sigmoid function, or gaussian radial basis function. Measures relevant size of hypothesis space, as with decision trees with k leaves. These bounds are tight for almost the entire range of parameters. Vapnik and chervonenkis, 1971, is not specific to neural networks. This is a somewhat surprising result, since purely linear networks result in vc dimension proportional. Neural networks with quadratic vc dimension 199 to w2 the proof relies on first showing that networks consisting of two types of activations, heavisides and linear, already have this power. To analyze the vc dimension, you need a system that can perfectly scatter a given data set.
Adversarial attacks during the testing phase of neural networks pose a challenge for the deployment. Combined with previous results, this gives an intriguing range of dependencies of the vcdimension on depth for networks with different nonlinearities. If the activation function is the sign function and the weights are general, then the vc dimension is at most. Nearlytight vcdimension and pseudodimension bounds for. Vapnikchervonenkis dimension of neural networks with binary. Vcdimension, neural networks, relu activation function extended abstract. The last reference, in particular, established for that case an upper bound of ow2, where, as before, w is the.
Vapnikchervonenkis dimension of recurrent neural networks article pdf available in discrete applied mathematics 861. No one seems to have pointed out in the above answers, that the vc dimension formula quoted is only for a 1layer neural network. Vcdimension of neural networks abbas mehrabian mcgill university adoiv postdoctoral ellfow 29 january 2019 1. Neural networks with quadratic vc dimension pascal koiran.
1451 1034 79 1273 1240 1242 341 531 73 81 399 1018 1513 52 1111 971 692 818 1180 1528 1441 610 66 28 393 1331 491 1454 1028 1006 1010 360 216 343 784 177