Can a doctor legally demand what is left of a prescription?

I was recently placed on Lortab for pain issues with my back (I paid about $90 for this medication). After 2 weeks it turns out that I needed to be put on something that lasted longer, so I was prescribed a drug called Opana. This medication worked very well and I was then given a prescription for a month. This afternoon out of the blue I received a call from the same doctors office DEMANDING I hand over to them what was left of the Lortab. I very politely told them this was not going to happen because, while I no longer needed the medication, I had paid for it myself and could find no good reason I should do this. I have no intent to do anything illegal/improper with these pills, but I was amazed this was even asked of me.