I would say the source is the most important but you are going to want something that is capable of reproducing it as accurately as possible. If the detail isn’t there to begin with, nothing is going to put it back in further down the chain. I’ve just upgraded my DAC and left the rest of the system unchanged. The difference is amazing. Obviously, the thing to do if you are buying or upgrading listen to the effect that improving different components has on the sound quality and buy accordingly.
I was really lucky. I had the Selekt with Katalyst DAC. I heard the Organik at the Linn factory and I had to have one. I got £500 trade in on my Katalyst and a Utopik power supply for free.