NEW COMPLEX VALUED ACTIVATIONFUNCTIONS: COMPLEX MODIFIEDSWISH, COMPLEX E-SWISH ANDCOMPLEX FLATTEN-TSWISH
Main Article Content
Abstract
Complex valued neural network (CVNN) has been developed to process complex valued data directly. In CVNN, one of the most important factors is selecting the node’s activation function. Choosing the right activation function for each layer is also crucial and may have a significant impact on metric scores and the training speed of the model. This paper introduces three new activation functions for CVNNs which is closely related to the activation function complex swish. These new activation functions are complex modified swish, complex E-swish and complex Flatten-T swish. In order to verify the validity and practicability of the proposed three new activation functions are tested and compared with complex swish activation function on complex valued four bit XOR problem, three inputs symmetry detection and the fading equalization problems. We show that complex E-swish ( β=1.4)  has the best overall performance when compared to other networks using complex swish, complex modified swish and complex Flatten-T swish activation functions on the considered tasks.
Downloads
Article Details
COPYRIGHT
Submission of a manuscript implies: that the work described has not been published before, that it is not under consideration for publication elsewhere; that if and when the manuscript is accepted for publication, the authors agree to automatic transfer of the copyright to the publisher.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work
- The journal allows the author(s) to retain publishing rights without restrictions.
- The journal allows the author(s) to hold the copyright without restrictions.
References
M. Ceylan, “Improving an algorithm with complex-valued artificial neural network and applications,†MS Thesis, Selcuk University Gradute School of Natural and Applied Sciences, Konya, 13-14, 2004.
A. Hirose, “Complex-Valued Neural Networks, World Scienific Publishing,†Singapore, 2003.
A. Hirose, “Complex-Valued Neural Networks, Series on Studies in Computational Intelligence,†Vol. 32, Springer-Verlag, New York 2006.
T. Nitta, “Complex-Valued Neural Networks: Utilizing High-Dimensional Parameters,†Information Science Reference, Hershey, PA, 2009.
T. Nitta, “Solving the XOR problem and the detection of symmetry using a single complex valued neuron,†Neural Networks, 16:1101-1105, 2003.
T. Nitta, “Orthogonality of Decision Boundaries in Complex-Valued Neural Networks,†Neural Computation 16, 73–97, 2004.
M. Celebi, M. Ceylan, “The New Activation Function for Complex Valued Neural Networks: Complex Swish Function,†ISAS WINTER-Samsun, 2019.
N. Özdemir, B. İskender, N. Özgür, “Complex valued neural network with Möbius activation function,†Commun Nonlinear Sci Numer Simulat 16, 2011.
H. Leung, S. Haykin, “The complex backpropagation algorithm,†IEEE Trans Signal Process,39:2101–4, 1991.
N. Benvenuto, F. Piazza, “On the complex backpropagation algorithm,†IEEE Trans Signal Process 40:967–9, 1992.
D.L. Birx, S.J. Pipenberg, “Chaotic oscillators and complex mapping feed forward networks (CMFFNS) for signal detection in noisy environments,†Proceedings of IEEE international jt. conf. neural networks. II, p. 881–888, 1992.
G. Kechriotis, E.S. Monalakos, “Training fully recurrent neural networks with complex weights,“ IEEE Trans Circuits Syst II, 41:235–8, 1994.
M. Kinouchi, M. Hagiwara, “Learning temporal sequences by complex neurons with local feedback,â€Proceedings of IEEE international conference on neural networks IV, p. 3165–3169,1995.
A. Hirose, “Applications of complex-valued neural networks to coherent optical computing using phase-sensitive detection scheme,†Inf Sci Appl, 2:103–17, 1994.
I. Aizenberg, “Complex-Valued Neural Networks with Multi-Valued Neurons, International Joint Conference on Neural Networks,†IJCNN, 2013.
N. Monning, S. Manandhar, “Evaluation of Complex-Valued Neural Networks on Real-Valued Classification Tasks,†CoRR abs/1811.12351, 2018.
P. Ramachandran, B. Zoph, “Searching for activation functions,†In International Conference on Learning Representations Q. V. L., 2018.
E. Alcaide, “E-swish: Adjusting activations to different network depths,†arXiv preprint arXiv:1801.07145, 2018.
H. Chieng, N. Wahid, O. Pauline, S. Perla, “Flatten-T Swish: a thresholded ReLU-Swish-like activation function for deep learning,†International Journal of Advances in Intelligent Informatics, 2018.
M. Ceylan, “Combined complex-valued artificial neural network (CCVANN),†Proceedings of the World Congress on Engineering. 2:955-959, 2011.
X. Chen, “An Modified Error Function for the Complex-value Backpropagation Neural Networks,†Neural Information Processing - Letters and Reviews Vol.8, No.1, 2005.
Y. Acar, M. Ceylan, E. Yaldız, “An examination on the effect of CVNN parameters while classifying the real-valued balanced and unbalanced data, International Conference on Artificial Intelligence and Data Processing (IDAP),†2018.
T. Nitta, “Orthogonality of Decision Boundaries in Complex-Valued Neural Networks,†Neural Computation 16, 73–97, 2004.
T. Nitta, “Complex-Valued Neural Network and Complex-Valued Backpropagation Learning Algorithm,†Advances in Imaging and Electron Physics,Volume 152, ISSN 1076-5670, 2008.
T. Nitta, “The Computational Power of Complex-Valued Neuron,†ICANN/ICONIP, 2003.
D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning internal representations by error propagation, Parallel Distributed Processing: Explorations in the microstructures of cognition,†vol. 1, pp. 318–362. Cambridge, MA: MIT Press, 1986.
D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning representations by back-propagating errors,†Nature 323, 533–536, 1986.
H.A. Jalab, R. W. Ibrahim, “New activation functions for complex-valued neural network,†International Journal of the Physical Sciences Vol. 6(7), pp. 1766-1772, 4 April, 2011.
T. Enomoto, K. Kakuda, S. Miura, “New Activation Functions in CNN and Its Applications,†ICCES, vol.1, no.2, pp.36-39, 2019.
M. Ceylan, “Combined complex-valued artificial neural network (CCVANN),†Proceedings of the World Congress on Engineering. 2:955-959, 2011.
H. Gürüler, M. Peker, “A Software Tool for Complex-Valued Neural Network: CV-ANN,†23nd Signal Processing and Communications Applications Conference (SIU), 2015.
X. Chen, Z. Tang C. Variappan, S. Li and T. Okada, “A modified error backpropagation algorithm for complex-value neural networks,†International Journal of Neural Systems, 15, 435-443, 2005.