Pinned content
View all 2 collectionsNLP admins have deemed these posts noteworthy.
Natural Language Processing FAQ
- reputation score 101
Can you answer these questions?
View all unanswered questionsThese questions still don't have an answer
Does my dual-KG Edge RAG make sense and how to Evaluate? Connecting 2 Knowledge Graphs. BA thesis
- reputation score 1
Inferencing in GLiNER model
- reputation score 1
Best pre-trained vision model for multi-plant disease detection in async web back-end
- reputation score 1
LDA Topic Modeling: Balancing Coherence Score (C_v) vs. Discrepant Downstream Predictor Importances
- reputation score 1763
Can I do a topic model for pre processing, find the topics I want and then do a second one?
- reputation score 1
Recommended answers
View all recommended answersThese answers have been recommended
Removing strange/special characters from outputs llama 3.1 model
- reputation score 123719
TL;DR Use this instead of rolling out your own detokenizer. tokenizer.batch_decode(input_ids) In Long The official Llama 3.1 has some approval process that might take some time, so this answer will ...
- reputation score 123719
Error while converting google flan T5 model to onnx
- reputation score 123719
Use https://huggingface.co/datasets/bakks/flan-t5-onnx instead. And to convert the google/flan-t5, see https://huggingface.co/datasets/bakks/flan-t5-onnx/blob/main/exportt5.py from pathlib import ...
- reputation score 123719
Why did my fine-tuning T5-Base Model for a sequence-to-sequence task has short incomplete generation?
- reputation score 822
Because of: labels = tokenizer(targets, max_length=32, padding="max_length", truncation=True) Most probably your model has learnt to just output/generate outputs that are ~32 tokens. Try: ...
- reputation score 123719
How to save the LLM2Vec model as a HuggingFace PreTrainedModel object?
- reputation score 123719
Wrapping the LLM2Vec object around like in https://stackoverflow.com/a/74109727/610569 We can try this: import torch.nn as nn from transformers import PreTrainedModel, PretrainedConfig from ...
- reputation score 123719
Mistral model generates the same embeddings for different input texts
- reputation score 123719
You're not slicing it the dimensions right at outputs.last_hidden_state[0, 0, :].numpy() Q: What is the 0th token in all inputs? A: Beginning of sentence token (BOS) Q: So that's the "embeddings&...
- reputation score 123719
See what's trending
View all trending questionsThese are the most active questions in NLP Collective
How does the Google "Did you mean?" Algorithm work? [closed]
- reputation score 1
spaCy: Can't find model 'en_core_web_sm'
- reputation score 26639
What is "entropy and information gain"? [closed]
- reputation score 890
How to compute the similarity between two text documents?
- reputation score 85310
googletrans stopped working with error 'NoneType' object has no attribute 'group'
- reputation score 856
Simply submit a proposal, get it approved, and publish it.
See how the process works