Because both deal with imaginary world building?

The distiction isn't clear anyway. Some "fantasy" book are more scientific than some "sci-fi" books - if they have a system of magic then is that any difference to FTL travel, or Vinge's Region's of Thought?