Questions about huggingface-tokenizers
Read more about huggingface-tokenizers
python (12.9k questions)
javascript (9.2k questions)
reactjs (4.7k questions)
java (4.2k questions)
java (4.2k questions)
c# (3.5k questions)
c# (3.5k questions)
html (3.3k questions)
Tokenizer didn't add BOS token when encoding the sentence
I would like to encode the sentence with BOS and EOS token. When I load a pretrained tokenizer, there is no BOS token, so I added BOS token to the tokenizer. After that, I encoded the sentence.
model_...
alryosha
Votes: 0
Answers: 1
ValueError: The state dictionary of the model you are trying to load is corrupted. Are you sure it was properly saved?
Goal: Amend this Notebook to work with albert-base-v2 model
Kernel: conda_pytorch_p36.
Section 1.2 instantiates a model from files in ./MRPC/ dir.
However, I think it is for a BERT model, not Albert. ...
DanielBell99
Votes: 0
Answers: 1
Optimize Albert HuggingFace model
Goal: Amend this Notebook to work with albert-base-v2 model
Kernel: conda_pytorch_p36.
Section 2.1 exports the finalised model. It too uses a BERT specific function. However, I cannot find an equivale...
DanielBell99
Votes: 0
Answers: 1
Chunked tokenization in huggingface has an arrow error
I'm following the code from this video at 1m25s, which shows:
def tokenize_and_chunk(texts):
return tokenizer(
texts["text"], truncation=True, max_length=context_length,
return ove...
Mittenchops
Votes: 0
Answers: 1