I think not wanting to travel with a crystal that nullifies your magic until you find a buyer who knows when or where is a pretty fair reason by itself tbh.

I imagine it’s something that could fetch a high price to the right buyer, but actually finding that buyer might not be trivial. How long would they be carrying around their own Kryptonite before they can pawn it off?

Also, one detail that did stand out to me here is the way Frieren seems a lot more casual about it, as someone with a much more academic interest in magic she views it as another novelty. But Fern seems a lot more genuinely terrified, as someone who feels like she needs magic to survive. She’s trembling when she can’t detect mana in the cave, and when Stark grabs her she starts fidgeting with her bracelet to try and calm her nerves.

And honestly, Fern being afraid is a good enough reason for Frieren and Stark not to push the matter any further.