We're not really disagreeing here? But wet paddy rice farming requires flat land, which in mountainous Japan is in notably short supply, so they planted other crops too.
This parallels China, where the warm, wet south is rice country but the colder, drier north grows other grains. The five grains (gokoku) idea is itself originally Chinese: