Exploiting heterogeneity in operational neural networks by synaptic plasticity
Date
2021Author
Kiranyaz, Mustafa SerkanMalik J.
Abdallah H.B.
Ince T.
Iosifidis A.
Gabbouj M.
...show more authors ...show less authors
Metadata
Show full item recordAbstract
The recently proposed network model, Operational Neural Networks (ONNs), can generalize the conventional Convolutional Neural Networks (CNNs) that are homogenous only with a linear neuron model. As a heterogenous network model, ONNs are based on a generalized neuron model that can encapsulate any set of non-linear operators to boost diversity and to learn highly complex and multi-modal functions or spaces with minimal network complexity and training data. However, the default search method to find optimal operators in ONNs, the so-called Greedy Iterative Search (GIS) method, usually takes several training sessions to find a single operator set per layer. This is not only computationally demanding, also the network heterogeneity is limited since the same set of operators will then be used for all neurons in each layer. To address this deficiency and exploit a superior level of heterogeneity, in this study the focus is drawn on searching the best-possible operator set(s) for the hidden neurons of the network based on the ?Synaptic Plasticity? paradigm that poses the essential learning theory in biological neurons. During training, each operator set in the library can be evaluated by their synaptic plasticity level, ranked from the worst to the best, and an ?elite? ONN can then be configured using the top-ranked operator sets found at each hidden layer. Experimental results over highly challenging problems demonstrate that the elite ONNs even with few neurons and layers can achieve a superior learning performance than GIS-based ONNs and as a result, the performance gap over the CNNs further widens.
Collections
- Electrical Engineering [2647 items ]
Related items
Showing items related by title, author, creator and subject.
-
Self-organized Operational Neural Networks with Generative Neurons
Kiranyaz, Mustafa Serkan; Malik J.; Abdallah H.B.; Ince T.; Iosifidis A.; Gabbouj M.... more authors ... less authors ( Elsevier Ltd , 2021 , Article)Operational Neural Networks (ONNs) have recently been proposed to address the well-known limitations and drawbacks of conventional Convolutional Neural Networks (CNNs) such as network homogeneity with the sole linear neuron ... -
Wireless Network Slice Assignment with Incremental Random Vector Functional Link Network
He, Yu Lin; Ye, Xuan; Cui, Laizhong; Fournier-Viger, Philippe; Luo, Chengwen; Huang, Joshua Zhexue; Suganthan, Ponnuthurai N.... more authors ... less authors ( IEEE Computer Society , 2022 , Article)This paper presents an artificial intelligence-assisted network slice prediction method, which utilizes a novel incremental random vector functional link (IRVFL) network to deal with the wireless network slice assignment ... -
A novel multi-hop body-To-body routing protocol for disaster and emergency networks
Ben Arbia, Dhafer; Alam, Muhammad Mahtab; Attia, Rabah; Ben Hamida, Elye ( Institute of Electrical and Electronics Engineers Inc. , 2016 , Conference Paper)In this paper, a new multi-hop routing protocol (called ORACE-Net) for disaster and emergency networks is proposed. The proposed hierarchical protocol creates an ad-hoc network through body-To-body (B2B) communication ...