Main Article Content

Jashanjot Kaur
Preetpal Kaur Buttar


Abstract: The research performed in this thesis is focused on the analysis of different methods to remove stopwords in Punjabi language. For organizing unstructured text in order to implement stopwords removal techniques, text preprocessing has to be applied. Text processing describes a variety of processing that is performed on raw data to prepare it for one more processing procedure which will be more helpful for performing some further, more purposeful analytic tasks. The words are called stopwords that occur most frequently in a document and contain very little information which is not essential in a document such as ਦੇ, ਹੈ, ਦੀ, ਤੇ, ਦਾ, ਨੇ, ਅਤੇ, ਤੋਂ etc. A list of such words is known as ‘stopwords list’ or ‘stopwords corpus’. These words are removed in the preprocessing phase of the text classification process.  The process of removing stopwords help to save time and reduces the size of those document. It also helps to increase the accuracy as well as performance of IR tasks. Most of the researchers worked on languages such as English, Arabic, Sanskrit etc. in the informational retrieval (IR) field. Therefore a lot of work and efforts need to be done in languages other than the languages in which the research has been already done to a great extent. The main goal of this thesis is to remove the stopwords in Punjabi language by using different techniques. Punjabi language is 11th  most-spoken language of India which is written in Gurmukhi script. Punjabi language is also used in mass media such as news, advertisements, movies, music etc. There is no standard stopword list created for Punjabi language as most of the stopwords lists are created for English and other languages. In this work, four different algorithms viz classical method using a pre-compiled stoplist, method based on frequency, method based on removing singletons and method based on Punjabi word corpus to remove Punjabi stopwords have been proposed, implemented and analyzed. Thus the size of the document is reduced by 30-35% by eliminating the set of such stopwords.


Download data is not yet available.

Article Details



H. P. Luhn, “A Statistical Approach to Mechanized Encoding and Searching of Literary Information,†IBM J. Res. Dev., vol. 1, no. 4, pp. 309–317, 1957.

R. Nisbet, J. elder, G. Miner, "Handbook of statistical analysis and data mining applications", academic Press, Elsevier, 2009.

A. Alajmi, E. M. Saad, R. R. Darwish, “Toward an ARABIC Stop-Words List Generationâ€, International Journal of Computer Applications (0975 – 8887) Volume 46– No.8, May 2012.

B. Alhadidi and M. Alwedyan, “Hybrid Stop-Word Removal Technique for Arabic Language.,†Egypt Comput Sci, vol. 30(1), no. 1, pp. 35–38, 2008.

C. Fox, A stop list for general text. ACM- SIGIR Forum, 24, 19-35, 1990.

R. B. Myerson, "Fundamentals of social choice theory", Discussion Paper No.1162, 1996.

C. T. Yuang, R. E. Banchs, C. E. Siong, “An Empirical Evaluation of Stop Word Removal in Statistical Machine Translationâ€, Journal of Computational Linguistics, 30-37, 2012.

Asubiaro, T. Victor, “ Entropy-Based Generic Stopwords List for Yoruba Textsâ€, International Journal of Computer and Information Technology (ISSN: 2279 – 0764) Volume 02– Issue 05, September 2013.

M. Sadeghi, J. Vegas, “Automatic identification of light stopwords for Persian information retrieval systemsâ€, Journal of Information Science, 1–12, 2014.

V. Jha, Manjunath N, P. Deepa Shenoy and Venugopal K R, “Hindi Stopword Removal Algorithm†International Conference on Microelectronics, Computing and Communication, MicroCom, 2016.