Stack Overflow
Stack Overflow

Questions tagged [bert-language-model]

BERT, or Bidirectional Encoder Representations from Transformers, is a new method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks.

The academic paper can be found here. And the original implementation of the BERT by google can be found here.

Reference

BERT Paper: https://arxiv.org/abs/1810.04805.

BERT Implementation: https://github.com/google-research/bert

909 questions
29
votes
7 answers

How to use Bert for long text classification?

We know that BERT has a max length limit of tokens = 512, So if an article has a length of much bigger than 512, such as 10000 tokens in text How can BERT be used?
user1337896
  • 551
  • 1
  • 5
  • 15
16
votes
5 answers

CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`

I got the following error when I ran my pytorch deep learning model in colab /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias) 1370 ret = torch.addmm(bias, input, weight.t()) 1371 else: ->…
Mr. NLP
  • 431
  • 1
  • 4
  • 17
16
votes
4 answers

How to cluster similar sentences using BERT

For ElMo, FastText and Word2Vec, I'm averaging the word embeddings within a sentence and using HDBSCAN/KMeans clustering to group similar sentences. A good example of the implementation can be seen in this short article:…
somethingstrang
  • 805
  • 2
  • 11
  • 24
12
votes
1 answer

PyTorch: RuntimeError: Input, output and indices must be on the current device

I am running a BERT model on torch. It's a multi-class sentiment classification task with about 30,000 rows. I have already put everything on cuda, but not sure why I'm getting the following run time error. Here is my code: for epoch in…
Roy
  • 181
  • 1
  • 1
  • 7
11
votes
2 answers

dropout(): argument 'input' (position 1) must be Tensor, not str when using Bert with Huggingface

My code was working fine and when I tried to run it today without changing anything I got the following error: dropout(): argument 'input' (position 1) must be Tensor, not str Would appreciate if help could be provided. Could be an issue with the…
Tashinga Musanhu
  • 121
  • 1
  • 3
11
votes
2 answers

Why Bert transformer uses [CLS] token for classification instead of average over all tokens?

I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further downstream task. Bert's last layer looks like this…
Aaditya Ura
  • 9,140
  • 4
  • 35
  • 62
10
votes
1 answer

BertForSequenceClassification vs. BertForMultipleChoice for sentence multi-class classification

I'm working on a text classification problem (e.g. sentiment analysis), where I need to classify a text string into one of five classes. I just started using the Huggingface Transformer package and BERT with PyTorch. What I need is a classifier with…
stackoverflowuser2010
  • 29,060
  • 31
  • 142
  • 184
9
votes
1 answer

PyTorch BERT TypeError: forward() got an unexpected keyword argument 'labels'

Training a BERT model using PyTorch transformers (following the tutorial here). Following statement in the tutorial loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels) leads to TypeError: forward() got an…
Anjani Dhrangadhariya
  • 741
  • 2
  • 6
  • 24
9
votes
2 answers

AttributeError: module 'torch' has no attribute '_six'. Bert model in Pytorch

I tried to load pre-trained model by using BertModel class in pytorch. I have _six.py under torch, but it still shows module 'torch' has no attribute '_six' import torch from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM #…
Ruitong LIU
  • 91
  • 1
  • 2
8
votes
2 answers

Training TFBertForSequenceClassification with custom X and Y data

I am working on a TextClassification problem, for which I am trying to traing my model on TFBertForSequenceClassification given in huggingface-transformers library. I followed the example given on their github page, I am able to run the sample code…
Rahul Goel
  • 634
  • 1
  • 5
  • 10
8
votes
2 answers

How to use Transformers for text classification?

I have two questions about how to use Tensorflow implementation of the Transformers for text classifications. First, it seems people mostly used only the encoder layer to do the text classification task. However, encoder layer generates one…
khemedi
  • 498
  • 7
  • 13
7
votes
1 answer

BERT embedding for semantic similarity

I earlier posted this question. I wanted to get embedding similar to this youtube video, time 33 minutes onward. 1) I dont think that the embedding that i am getting from CLS token are similar to what is shown in the youtube video. I tried to…
user2543622
  • 4,682
  • 20
  • 62
  • 117
7
votes
2 answers

Get probability of multi-token word in MASK position

It is relatively easy to get a token's probability according to a language model, as the snippet below shows. You can get the output of a model, restrict yourself to the output of the masked token, and then find the probability of your requested…
Bram Vanroy
  • 22,919
  • 16
  • 101
  • 195
6
votes
2 answers

BertModel transformers outputs string instead of tensor

I'm following this tutorial that codes a sentiment analysis classifier using BERT with the huggingface library and I'm having a very odd behavior. When trying the BERT model with a sample text I get a string instead of the hidden state. This is the…
Miguel
  • 1,804
  • 2
  • 22
  • 42
6
votes
1 answer

HuggingFace BERT `inputs_embeds` giving unexpected result

The HuggingFace BERT TensorFlow implementation allows us to feed in a precomputed embedding in place of the embedding lookup that is native to BERT. This is done using the model's call method's optional parameter inputs_embeds (in place of…
Vivek Subramanian
  • 934
  • 1
  • 13
  • 23
1
2 3
…
60 61