But for a Jedi, the lightsaber becomes more: it is a manifestation of a Jedi's connection to the Force.' ― Jaden Korr
And while a non-Force user could probably ignite and wield a lightsaber, the crystal is needed to be properly attuned to the Force. Without that attunement, the crystal is just a rock. Like the Force user, the crystal is attuned to the Force. ' The crystal is, by itself, the power source of the weapon.