Because a non-violent tribe will be destroyed by any violent neighbors it has. Even if it had a protected group of non-violent members, it would be militarily disadvantaged compared to its enemies with more fighters per population. And what are those non-violent people even doing all day in a world without any technological progress? Don't imagine you'd invent technology because almost nobody did much of that. If it was truly isolated like the Chatham Island Maori then yea, but they were eventually genocided by the mainland New Zealand Maori when they finally got to them.

I think you would be conscripted to fight whether you liked it or not, and inculcated with the belief that it's the right thing to do. Maybe even feeling shame are your unwillingness but forcing yourself to fit in anyway.