STOPWORDS REMOVAL AND ITS ALGORITHMS BASED ON DIFFERENT METHODS
Main Article Content
Abstract
Abstract: The research performed in this thesis is focused on the analysis of different methods to remove stopwords in Punjabi language. For organizing unstructured text in order to implement stopwords removal techniques, text preprocessing has to be applied. Text processing describes a variety of processing that is performed on raw data to prepare it for one more processing procedure which will be more helpful for performing some further, more purposeful analytic tasks. The words are called stopwords that occur most frequently in a document and contain very little information which is not essential in a document such as ਦੇ, ਹੈ, ਦੀ, ਤੇ, ਦਾ, ਨੇ, ਅਤੇ, ਤੋਂ etc. A list of such words is known as ‘stopwords list’ or ‘stopwords corpus’. These words are removed in the preprocessing phase of the text classification process. The process of removing stopwords help to save time and reduces the size of those document. It also helps to increase the accuracy as well as performance of IR tasks. Most of the researchers worked on languages such as English, Arabic, Sanskrit etc. in the informational retrieval (IR) field. Therefore a lot of work and efforts need to be done in languages other than the languages in which the research has been already done to a great extent. The main goal of this thesis is to remove the stopwords in Punjabi language by using different techniques. Punjabi language is 11th most-spoken language of India which is written in Gurmukhi script. Punjabi language is also used in mass media such as news, advertisements, movies, music etc. There is no standard stopword list created for Punjabi language as most of the stopwords lists are created for English and other languages. In this work, four different algorithms viz classical method using a pre-compiled stoplist, method based on frequency, method based on removing singletons and method based on Punjabi word corpus to remove Punjabi stopwords have been proposed, implemented and analyzed. Thus the size of the document is reduced by 30-35% by eliminating the set of such stopwords.
Downloads
Article Details
COPYRIGHT
Submission of a manuscript implies: that the work described has not been published before, that it is not under consideration for publication elsewhere; that if and when the manuscript is accepted for publication, the authors agree to automatic transfer of the copyright to the publisher.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work
- The journal allows the author(s) to retain publishing rights without restrictions.
- The journal allows the author(s) to hold the copyright without restrictions.
References
H. P. Luhn, “A Statistical Approach to Mechanized Encoding and Searching of Literary Information,†IBM J. Res. Dev., vol. 1, no. 4, pp. 309–317, 1957.
R. Nisbet, J. elder, G. Miner, "Handbook of statistical analysis and data mining applications", academic Press, Elsevier, 2009.
A. Alajmi, E. M. Saad, R. R. Darwish, “Toward an ARABIC Stop-Words List Generationâ€, International Journal of Computer Applications (0975 – 8887) Volume 46– No.8, May 2012.
B. Alhadidi and M. Alwedyan, “Hybrid Stop-Word Removal Technique for Arabic Language.,†Egypt Comput Sci, vol. 30(1), no. 1, pp. 35–38, 2008.
C. Fox, A stop list for general text. ACM- SIGIR Forum, 24, 19-35, 1990.
R. B. Myerson, "Fundamentals of social choice theory", Discussion Paper No.1162, 1996.
C. T. Yuang, R. E. Banchs, C. E. Siong, “An Empirical Evaluation of Stop Word Removal in Statistical Machine Translationâ€, Journal of Computational Linguistics, 30-37, 2012.
Asubiaro, T. Victor, “ Entropy-Based Generic Stopwords List for Yoruba Textsâ€, International Journal of Computer and Information Technology (ISSN: 2279 – 0764) Volume 02– Issue 05, September 2013.
M. Sadeghi, J. Vegas, “Automatic identification of light stopwords for Persian information retrieval systemsâ€, Journal of Information Science, 1–12, 2014.
V. Jha, Manjunath N, P. Deepa Shenoy and Venugopal K R, “Hindi Stopword Removal Algorithm†International Conference on Microelectronics, Computing and Communication, MicroCom, 2016.