"I was reading through the book of Galatians the other day and came across verse 4.10-11 [You are observing special days, and months, and seasons, and years. I am afraid that my work for you may have been wasted.] It seems Paul is saying we're not supposed to celebrate holidays. Is this true?"
When Paul wrote to the Galatian church, he was writing to a troubled people. Galatia was largely a non-Jewish populated region and the church of the Galatians had little Jewish influence--at least when it was founded. However, sometime between its founding and this letter Jewish Christians had become a part of the congregation, or had exerted some influence, and began teaching that to be a legitimate Christian one had to become a practicing Jew as well.
It is important to remember that it wasn't Jesus' intentions to begin a new religion. His emphasis was in a reformation of the Jewish faith. His message that the "law was fulfilled" did not mean the law (the rules necessary to obey to be a practicing Jew) was no longer relevant, but that in its present practice it had become oppressive ("Do not think that I have come to abolish the law or the prophets . . . For truly I tell you, until heaven and earth pass away, not one letter, not one stroke of a letter, will pass from the law" (Matthew 5.17-18)). However, by the time Paul began his evangelizing, it was clear the Jewish faith had rejected both Jesus' reformation movement and messianic claim and Christianity became a faith unto itself.
The difficulty arose in defining just what the Christian faith was. For Jewish-Christians, it was clear Jesus was trying to reform Judaism, so they clung to the notion that a convert had to obey all the Jewish law as well. However, some, including Paul, taught that being a Christian meant obeying the teachings of Christ alone. This is the doctrine taught to the Galatian church at its inception, but others were teaching the Galatians they had to also obey all the Jewish laws.
This, then, is the background of the above passage. The "special days, months, seasons, and years" all relate to the Jewish religious holidays and laws. Paul was rejecting the notion that the Galatians had to observe the Jewish laws and customs to be Christians. He rejected the law that the Sabbath was any more "holy" than any other day which God had made. He rejected the special holidays of ritual purification, the seasons of fasting, and the agricultural year set apart as holy--not because it was wrong to celebrate a holiday, but because it was not necessary to observe the Jewish laws in order to be Christian, which was the contention.
It is not necessary to keep Yom Kippur nor Chanukah to be a Christian. Neither is it necessary to keep Christmas or Easter. The fact is, being a Christian (and indeed, being a Jew) is more than the observance of rote customs and practices prescribed by rules and laws. Paul wasn't trying to keep the Galatians from having holidays or celebrations, but to keep them from observing these holidays because they were somehow "required" to do so to be deemed righteous. The commandments of loving God and loving neighbor outweigh all other laws, so if celebrating Christmas, Easter, birthdays, the Fourth of July, Yom Kippur, or Chanukah demonstrates your love for either God or your neighbor, then by all means, celebrate!