@JonathanAgresta Are they replying from the email notification or are they actually seeing the word "reply" in the announcement page?
I just tested it and on the announcement page, I could only reply to ones with comments turned on. I tested both with and without the new Announcements/Discussion feature turned on.
They are replying via email and those emails are posting as replies to the announcements. It's a problem since they discuss their private grades. Disabling comments used to prevent this.
I have recently experienced exactly the same issue as described by @JonathanAgresta. I always disable comments on Announcements, so I was very surprised to see that a student had posted a comment to one of my Announcements. It happened three or four days ago. I double-checked my Course Settings, and there were fine.