You can disable the automatic handoff in the Apple TV settings. That drove me crazy as well.