Personally I think Caste Systems are gay because:
There is never any guarantee that a child born to the Manual Labor caste will have nothing to offer the world besides manual labour.
There's never a guarantee that a child of the Ruling Caste will actually be good at ruling.
Isn't giving one type of person more rights than others for being born into a "more correct" caste basically Affirmative Action Diversity-Hire bullshit?
What guarantee a "Military Caste" of people forced into serving the military will be as good as/better than a military where people from all walks of life can try to enter but only the best of the best are allowed to serve?
You never know what someone will be good at until after they've had chances to try everything and learn what they can and can't do.
How many innovations and improvements and discoveries vital to the good shit we have today were made by "special people" given the absolute best education, and how many of those were made by people who came from pretty much nothing yet still distinguished themselves through their accomplishments and rose up without the need of a caste system's artificial elevation?
How can anyone justify denying someone in the worker caste the right to study something in his free time that could be used to create his own business, creating more jobs and economic competition?
How many generations would it take for those castes to culturally and evolutionarily diverge from one another to the point where they wouldn't want to interbreed, whether the laws forbid it or not
How bad do things have to get for the "Sucks To Be These Guys caste" before they're ready to cause problems the second someone tells them to and promises them a better life?
Tyrants love fracturing and dividing society to try and make different groups hate each other, so they won't unite against the tyrant. Tyrants would love caste systems, because they neatly divide society for those Tyrants ahead of time!
And while we're on the subject, why do so many novels have idealized caste systems pushed to such absurd extremes that workers are taxed to directly pay for the absurdly expensive bullshit loved by the do-nothing upper classes? They literally just get paid for nothing, it's ridiculous! Sure you could say "Rich idiots buy more stuff therefore economic activity" but such an economy would morph to suit the expensive tastes of rich idiots whose needs are already met, while practically abandoning other castes/putting them on the back burner because there's less money in them per sale. That might be a good way to keep rich idiots pacified and pro-government but what future would a society dedicated to the fleeting pleasures of the lucky fickle idle rich have?