Description
Classify IMDB reviews in negative and positive categories.
Predicted Entities
neg, pos
How to use
document_assembler = DocumentAssembler() \
.setInputCol("text") \
.setOutputCol("document")
sentencer = SentenceDetectorDLModel().pretrained("sentence_detector_dl", "en")\
.setInputCols("document")\
.setOutputCol("sentence")
tokenizer = Tokenizer().setInputCols('sentence').setOutputCol('tokens')
embeddings = WordEmbeddingsModel().pretrained("glove_840B_300", "xx")\
.setInputCols(['document','tokens']).setOutputCol('word_embeddings')
sentence_embeddings = SentenceEmbeddings() \
.setInputCols(["document", "word_embeddings"]) \
.setOutputCol("sentence_embeddings") \
.setPoolingStrategy("AVERAGE")
classifier = SentimentDLModel().pretrained('sentimentdl_glove_imdb')\
.setInputCols(["sentence_embeddings"])\
.setOutputCol("sentiment")
nlp_pipeline = Pipeline(stages=[document_assembler,
sentencer, tokenizer,
embeddings, sentence_embeddings,
classifier
])
l_model = LightPipeline(nlp_pipeline.fit(spark.createDataFrame([['']]).toDF("text")))
annotations = l_model.fullAnnotate('Demonicus is a movie turned into a video game! I just love the story and the things that goes on in the film.It is a B-film ofcourse but that doesn`t bother one bit because its made just right and the music was rad! Horror and sword fight freaks,buy this movie now!')
Results
| | document | sentiment |
|---:|---------------------------------------------------------------------------------------------------------:|--------------:|
| | Demonicus is a movie turned into a video game! I just love the story and the things that goes on in the | |
| 0 | film.It is a B-film ofcourse but that doesn`t bother one bit because its made just right and the music | positive |
| | was rad! Horror and sword fight freaks,buy this movie now! | |
Model Information
| Model Name: | sentimentdl_glove_imdb |
| Compatibility: | Spark NLP 2.7.1+ |
| License: | Open Source |
| Edition: | Official |
| Input Labels: | [sentence_embeddings] |
| Output Labels: | [sentiment] |
| Language: | en |
| Dependencies: | glove_840B_300 |
Data Source
https://ai.stanford.edu/~amaas/data/sentiment/
Benchmarking
precision recall f1-score support
neg 0.85 0.85 0.85 12500
pos 0.87 0.83 0.85 12500
accuracy 0.84 25000
macro avg 0.86 0.84 0.85 25000
weighted avg 0.86 0.84 0.85 25000