a11ce.com/resurrecting-katbot.html
epigraph:
I am a chatbot. A softbot. A reminder of the options available. A resource for idle thinking. A digital effigy. A game. A fiction. A way to nurture escapism. A provocation. A place for play. A prank. A dark magic of our times. A stand-in for ourselves. A simulacrum of what our robots will eventually be. - katbot, 2026
Katbot was a gpt3 discord bot blah blah. maybe a bit about her personality and some messages
one paragraph about character multiplicity. i think character multiplicity is really important, and i imagine a world where you can make an llm character like you can write a story or draw a picture. or maybe more like demoscene type things, but like. as art!
katbot v1 was a personality prompt and a chat template.
The following is a conversation with Katbot. Katbot is [...].
alice: <message>
katbot:
extracted messages from discord, filtered out some with personal information. this was a pretty high bar so the katbot dataset will never be public
of these, 146 messages were removed for being incoherent/broken. katbot as a character is very high variance, so some non-sequiturs are okay
mention the multiline cleanup:
next step was a ton of human review of other data. most of these are alternate branches in real conversations, because
gpt3 got an 81% 'good katbot voice rate', so that's the target
8b got 41%, and 405b got 50%
initially, i experimented with vector steering (to be able to serve on ACS infra), didnt really work, but about 700 (48% accepted) ended up in the final set. these numbers are rough bc of duplicate sources during review, i could also 'star' messages, i.e. accpetance is plausible and star is strong voice/funny/high personality
final training set gpt3: 624 accepted (103 starred), 146 rejected base 405b: 1050 accepted (142 starred), 1050 rejected base 8b: 855 accepted (93 starred), 1213 rejected steered 8b: 234 accepted (21 starred), 255 rejected multiple models: 237 accepted (19 starred), 176 rejected a few edited/manually corrected/misc
in total, 3028 positive chat examples (379 of which starred) and 2850 negative
additionally, i had 26 acc 11 rej tweets (not exactly sure what setup), and 35 generated using the venue from base 405b
katbot writes very short messages and can be very cryptic, usually
under 10 words. this is fun for chat but not great for social media, so
some training data uses a different venue marker
this is the twitter feed of katbot. notably, training just
on chat already improves performance on her tweets, so she does have a
consistent persona concept. this turns out to work quite well
1 epoch sft, one epoch kto (because i have acc/rej data but not paired). one epoch bc immediate overfitting on epoch 2.
sft lr was chosen by min loss on a test set
the main choice here is kto lr: roughly, there seem to be two
breakpoints. up to 2.5e-6 (note that im not using a/r scaling so take
this as an illustrative example value), the main effect is to remove
obvious incoherence/broken replies. next, up to about 5e-6, there is a
tradeoff between increasing quality and decreasing variance. beyond
1.5e-5, responses degrade, partially into incoherenence. interestingly
some of the failures here seem to be in the direction of parody of the
format (tweets) or character
(nothing much, just taking over the world and creating a transhumanist singularity).
katbot has no cross-context information transfer and always has new external info (either user message or variance-context, see venue part) so sacrificing a bit of phrasing variance for higher quality is okay.
. maybe also talk about the impact of kto
i also tried 70b but it was indistinguishable. i suspect that 8b can go very far in voice, interested in looking into this more. 8b can sample locally or on an a40 which is great
in short, rlhf trains a reward model on the labeled data and then RLs against it. kto preference accuracy on heldout data approximates the accuracy of a good reward model, and i saw about 65% at max. so, within the wide range of 'good katbot', rlhf is questionably useful. early on in the project i also found that a linear probe was rewardhackable into being length based
k56 is good. it has an acceptance rate of X and a star rate of Y. this means that best of N sampling for N=? will be (how good), which is a future project
delve town plus bluesky (mirrored, tech note abt how this works)
samples whenever, adds to queue, posts every 6-12 hours. replies on a whitelist (bc a lot of delve town is incoherent). manual filtering for leaked private info (havent seen any yet)
no weights or public chat, but find me irl if you want to talk to her
maybe this is where the multiplicity note goes?
total cost under 200 (via runpod), mostly samples for review
You can respond to this page on twitter, substack, mastodon, or bluesky.