Ok, so answering OP’s question, what is actually changing during burn-in in a speaker?

Yes, the effect is measurable, but I read the question as being “what’s changed in my speaker?”

From your final para, it sounds like it’s not the driver because putting a new one made no difference.