python (12.9k questions)

javascript (9.2k questions)

reactjs (4.7k questions)

java (4.2k questions)

java (4.2k questions)

c# (3.5k questions)

c# (3.5k questions)

html (3.3k questions)

Questions - tokenize

bert_vocab.bert_vocab_from_dataset taking too long

I'm following this tutorial (https://colab.research.google.com/github/tensorflow/text/blob/master/docs/guide/subwords_tokenizer.ipynb#scrollTo=kh98DvoDz7Jn) to generate a vocabulary from a custom data...
test-img

Kurt

tensorflow

tokenize

bert-language-model

Votes: 0

Answers: 1

Latest Answer

I ran into the same issue. This is how I resolved it: First I checked the number of elements in the dataset: examples, metadata = tfds.load('my_dataset', as_supervised=True, with_info=True) print(meta...
test-img

rebo

how to replace the command tfds.load for imdb reviews with download dataset file?

tfds.load(name="imdb_reviews", data_dir=direc, split="train", with_info=True, as_supervised=True) i have download the dataset , it has downloads and imdb_reviews directories, in th...
test-img

lucky

python

nlp

tokenize

tensorflow-datasets

Votes: 0

Answers: 1

Latest Answer

I found the answer. if you give the directory to the command tfds.load() then the next time it does not download the dataset because it finds out there exits the data in your drive. so there is actual...
test-img

lucky

tokenize the url

I have a sentences (logs) look like that ['GET http://10.0.0.0:1000/ HTTP/X.X'] I want to have it in this form : ['GET', 'http://10.0.0.0:1000/', 'HTTP/X.X'] but that's not the fall. i've used this ...
test-img

Hermoine

python

tokenize

Votes: 0

Answers: 2

Latest Answer

import re sentences = ['GET http://10.0.0.0:1000/ HTTP/X.X'] words=[] for sent in sentences: words.append(list(sent.split(' '))) print(words) Can you use a simple space split? I think nltk.wo...
test-img

Krushil Dhamelia

Problem while creating the NLP model using tensorflow?

I have created a spam classifier model using tensorflow.. ( tokenization , embedding layer and lstm ). model=Sequential() model.add(Embedding(voc_size,32,input_length=sent_len)) model.add(LSTM(units=1...
test-img

Nitish Pandey

python

tensorflow

nlp

tokenize

word-embedding

Votes: 0

Answers: 0

Posts

Questions

Blogs

Jobs