Honestly I think you are right. If it were obvious votes were public people would act differently

Why? Lemmy is a pseudonymous platform. Who cares what content individual users up or downvote. The votes being public is the only way to deal with brigading across instances.