python (12.9k questions)

javascript (9.2k questions)

reactjs (4.7k questions)

java (4.2k questions)

java (4.2k questions)

c# (3.5k questions)

c# (3.5k questions)

html (3.3k questions)

Questions - apache-spark-ml

How to get vocabulary size of word2vec?

I have a pretrained word2vec model in pyspark and I would like to know how big is its vocabulary (and perhaps get a list of words in the vocabulary). Is this possible? I would guess it has to be store...
test-img

Luca Clissa

machine-learning

pyspark

nlp

word2vec

apache-spark-ml

Votes: 0

Answers: 1

Latest Answer

Not quite sure why you doubt the result of .getVectors().count(), which gives the desired result indeed, as shown in the documentation link you have provided yourself. Here is the example posted there...
test-img

desertnaut

How to troubleshoot this PySpark GLM error?

Trying to run a GLM using poisson family and log link function and getting the following errors: 2022-01-11 15:56:55,143 root ERROR An error occurred while calling o266.fit. : java.lang.NullPointerExc...
test-img

Evan Zamir

apache-spark

pyspark

regression

glm

apache-spark-ml

Votes: 0

Answers: 1

Latest Answer

It turns out I ran into this issue with Spark 5 years ago and forgot about it. In fact I even posted a bug report that was never fixed (because supposedly it is the "correct" behavior). The ...
test-img

Evan Zamir

Remove specific stopwords Pyspark

New to Pyspark, I'd like to remove some french stopwords from pyspark column. Due to some constraint, I can't use NLTK/Spacy, StopWordsRemover is the only option that I got. Below is what I have tried...
test-img

A2N15

python

apache-spark

pyspark

apache-spark-ml

stop-words

Votes: 0

Answers: 2

Latest Answer

The problem is that you have trailing spaces in your stop words. Also, you don't need to lowercase them unless you need the StopWordsRemover to be case sensitive. By default it is set to false, you ca...
test-img

blackbishop

How best to shuffle columns in pyspark to calculate permutation feature importance

It seems that pyspark ML doesn't have a built-in permutation feature importance method. So I want code this up, and to do so I have to individually shuffle each column in the dataframe. I found this r...
test-img

Gaurav Bansal

apache-spark

pyspark

apache-spark-sql

shuffle

apache-spark-ml

Votes: 0

Answers: 1

Latest Answer

Spark dataframes are unordered, so these types of transformations will always be expensive. You might want to consider converting into pandas to do the shuffle part then convert back again to pyspark:...
test-img

blackbishop

Posts

Questions

Blogs

Jobs