Top-p sampling
Language model technique
Top-p sampling, also known as nucleus sampling, is a stochastic decoding strategy for generating sequences from autoregressive probabilistic models. It was originally proposed by Ari Holtzman, Yejin Choi and colleagues in 2019 for natural language generation to address the issue of repetitive and nonsensical text generated by other common decoding methods like beam search.
Nº Q122237668 ★
Common · Knowledge
Top-p sampling
Language model technique
Top-p sampling, also known as nucleus sampling, is a stochastic decoding strategy for generating sequences from autoregressive probabilistic models. It was originally proposed by Ari Holtzman, Yejin Choi and colleagues in 2019 for natural language generation to address the issue of repetitive and nonsensical text generated by other common decoding methods like beam search.
From Wikipedia
Top-p sampling, also known as nucleus sampling, is a stochastic decoding strategy for generating sequences from autoregressive probabilistic models. It was originally proposed by Ari Holtzman, Yejin Choi and colleagues in 2019 for natural language generation to address the issue of repetitive and nonsensical text generated by other common decoding methods like beam search. The technique has since been applied in other scientific fields, such as protein engineering and geophysics. In top-p sampling, a probability threshold p is set, and the next item in a sequence is sampled only from the smallest possible set of high-probability candidates whose cumulative probability exceeds p. This method adapts the size of the candidate pool based on the model's certainty, making it more flexible than top-k sampling, which samples from a fixed number of candidates. Due to its effectiveness, top-p sampling is a widely used technique in many large language model applications.
Text: Wikipédia, CC BY-SA 4.0. ·
Related cards
-
Gibbs sampling
Algorithm
Nº Q1191905 ★
Not listed
-
Bottom-up and top-down approaches
Strategies of information processing and ordering of knowledge
Nº Q2167279 ★★
Not listed
-
PP (complexity)
Complexity class
Nº Q1563053 ★
Not listed
-
Topic model
Type of model
Nº Q3532085 ★
Not listed
-
S
Stochastic parrot
Metaphor to describe the theory that large language models, though able to generate plausible language, do not understand the meaning of the language they process
Nº Q122663230 ★★★
Not listed
-
GPT-1
Generative pre-trained transformer-based language model from 2018
Nº Q95726718 ★★
Not listed