If you read the text carefully - (Exodus 32), you will find that only 3000 people (verse 28) sinned with the Golden Calf. Even so, yes indeed, this is the most serious transgression in Jewish History, to the extent that we are STILL paying the price for it, 3317 years later. Also from the text (verse 1), Aaron, was not the source of the initiative. These people came to him.

The following is an interesting review of Golden Calf incident.