My new favorite LLM bug that I've seen is that sometimes when the context gets too long with DeepSeek R1 the model starts spitting out random pieces of training data in Japanese. It'll be in the middle of generating a diff for for some code and it just starts spitting out Kanji in the middle of it. My favorite part is that the model itself seems super confused and during its thinking process says stuff like "oh my it looks like I didn't generate that diff correctly, let's try that again and do exactly what's in the file"
I (unsurprisingly) can't read Kanji but I tried Googling some of the stuff and it seems like the little inserted bits of text are excerpts from Japanese government or university websites?