I don't think the linguists would consider arabic numerals on their own to be a language either. The main distinction, as I understand it, is having something like a grammar, i.e. a set of consistent rules about how to arrange the symbols to have meaning beyond just the sum of the meaning of each individual symbol. So no matter how you mark down your count, it's not language until you have some consistent pattern of signifying that that means how many moons have passed, or how many people are in the local community, or something like that.