Accéder directement au contenu Accéder directement à la navigation

An Embedding of ReLU Networks and an Analysis of their Identifiability

2 DANTE - Dynamic Networks : Temporal and Structural Capture Approach
Inria Grenoble - Rhône-Alpes, LIP - Laboratoire de l'Informatique du Parallélisme, IXXI - Institut Rhône-Alpin des systèmes complexes
Abstract : Neural networks with the Rectified Linear Unit (ReLU) nonlinearity are described by a vector of parameters $\theta$, and realized as a piecewise linear continuous function $R_{\theta}: x \in \mathbb R^{d} \mapsto R_{\theta}(x) \in \mathbb R^{k}$. Natural scalings and permutations operations on the parameters $\theta$ leave the realization unchanged, leading to equivalence classes of parameters that yield the same realization. These considerations in turn lead to the notion of identifiability -- the ability to recover (the equivalence class of) $\theta$ from the sole knowledge of its realization $R_{\theta}$. The overall objective of this paper is to introduce an embedding for ReLU neural networks of any depth, $\Phi(\theta)$, that is invariant to scalings and that provides a locally linear parameterization of the realization of the network. Leveraging these two key properties, we derive some conditions under which a deep ReLU network is indeed locally identifiable from the knowledge of the realization on a finite set of samples $x_{i} \in \mathbb R^{d}$. We study the shallow case in more depth, establishing necessary and sufficient conditions for the network to be identifiable from a bounded subset $\mathcal X \subseteq \mathbb R^{d}$.
Type de document :
Article dans une revue
Domaine :

https://hal.archives-ouvertes.fr/hal-03292203
Contributeur : Pierre Stock Connectez-vous pour contacter le contributeur
Soumis le : lundi 3 janvier 2022 - 19:05:26
Dernière modification le : vendredi 21 janvier 2022 - 04:07:35

Fichier

main.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

• HAL Id : hal-03292203, version 2
• ARXIV : 2107.09370

Citation

Pierre Stock, Rémi Gribonval. An Embedding of ReLU Networks and an Analysis of their Identifiability. Constructive Approximation, Springer Verlag, In press. ⟨hal-03292203v2⟩

Métriques

Les métriques sont temporairement indisponibles