Language Detection & Identification Pipeline - 95 Languages

Description

Language detection and identification is the task of automatically detecting the language(s) present in a document based on the content of the document. LanguageDetectorDL is an annotator that detects the language of documents or sentences depending on the inputCols. In addition, LanguageDetetorDL can accurately detect language from documents with mixed languages by coalescing sentences and select the best candidate.

We have designed and developed Deep Learning models using CNN architectures in TensorFlow/Keras. The models are trained on large datasets such as Wikipedia and Tatoeba with high accuracy evaluated on the Europarl dataset. The output is a language code in Wiki Code style: https://en.wikipedia.org/wiki/List_of_Wikipedias.

This pipeline can detect the following languages:

Afrikaans, Amharic, Aragonese, Arabic, Assamese, Azerbaijani, Belarusian, Bulgarian, Bengali, Breton, Bosnian, Catalan, Czech, Welsh, Danish, German, Greek, English, Esperanto, Spanish, Estonian, Basque, Persian, Finnish, Faroese, French, Irish, Galician, Gujarati, Hebrew, Hindi, Croatian, Haitian Creole, Hungarian, Armenian, Interlingua, Indonesian, Icelandic, Italian, Japanese, Javanese, Georgian, Kazakh, Khmer, Kannada, Korean, Kurdish, Kyrgyz, Latin, Luxembourgish, Lao, Lithuanian, Latvian, Malagasy, Macedonian, Malayalam, Mongolian, Marathi, Malay, Maltese, Nepali, Dutch, Norwegian Nynorsk, Norwegian, Occitan, Odia (Oriya), Punjabi (Eastern), Polish, Pashto, Portuguese, Quechua, Romanian, Russian, Northern Sami, Sinhala, Slovak, Slovenian, Albanian, Serbian, Swedish, Swahili, Tamil, Telugu, Thai, Tagalog, Turkish, Tatar, Uyghur, Ukrainian, Urdu, Vietnamese, Volapük, Walloon, Xhosa, Chinese.

Open in Colab Download

How to use

from sparknlp.pretrained import PretrainedPipeline

pipeline = PretrainedPipeline("detect_language_95", lang = "xx")

pipeline.annotate("French author who helped pioneer the science-fiction genre.")
import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline

val pipeline = new PretrainedPipeline("detect_language_95", lang = "xx)

pipeline.annotate("French author who helped pioneer the science-fiction genre.")
import nlu

text = ["French author who helped pioneer the science-fiction genre."]
lang_df = nlu.load("xx.classify.lang.95").predict(text)
lang_df

Results

{'document': ['French author who helped pioneer the science-fiction genre.'],
 'language': ['en']}

Model Information

Model Name: detect_language_95
Type: pipeline
Compatibility: Spark NLP 2.7.0+
Edition: Official
Language: xx

Included Models

  • DocumentAssembler
  • SentenceDetectorDLModel
  • LanguageDetectorDL