1. X
  2. kyokopom
Log inSign up
kyokopom
4,983 posts
user avatar
kyokopom
@Kyokopom
Investigador independiente de Modelos pequeños de IA, vtuberMex, VtuberESP, Trainer LLM, actualmente estoy desarrollando mi propio modelo mini
Mexico
Joined September 2024
173
Following
73
Followers
RepliesRepliesMediaMedia

Log in or sign up for X

See what’s happening and join the conversation

Continue with phone
or
Log in with username or email
Terms·Privacy·Cookies·Accessibility·Ads Info·© 2026 X Corp.
  • Pinned
    user avatar
    kyokopom
    @Kyokopom
    Aug 28, 2025
    Right now my goal is to develop the first open-source model trained in FP4 using PyTorch primitives and contribute to its development. I know it’s tough, but it’s going to be a journey where I’ll need a lot of help. The first step would be training NanoGPT on an RTX 5090. The
    1.8K
  • user avatar
    kyokopom
    @Kyokopom
    17h
    mile loss (eval 25) CE (eval 20)
    18
  • user avatar
    kyokopom
    @Kyokopom
    Jul 15
    Hi @zhangkechi, in the released StackTrans code, the final token’s stack state seems to be reused across all positions in the next layer. Is this intentional? Could it violate autoregressive causality, or was another causal implementation used for the paper?
    40
  • user avatar
    kyokopom
    @Kyokopom
    Jul 12
    waos
    16
  • user avatar
    kyokopom
    @Kyokopom
    Jul 10
    After a few tests, I found a small improvement to CONDA that removes the fixed scaling from the paper. It made training only (5%) slower with no clear gain over the baseline, but it suggests CONDA still has room to improve.
    33