I grew up in the "Gifted and Talented" program in my hometown of Paterson, NJ. Our teachers did the best job that they could in challenging us, but it was implemented poorly: a 35+ student class is likely to vary wildly in terms of ability. The best moments were breaking up into small groups of 4-5 students and working on sessions individually. This is something that I've carried into my larger tutoring sessions.
What happened to many of us in the Gifted and Talented traditional class structure? We got bored. We started to screw around. I saw plenty of good students completely fall off of the wagon. The curiosity and challenge that needs to be instilled in many of our future scientists was squashed when faced with an educational situation that doesn't challenge them. I was lucky enough to have a good support system at home to reinforce lessons, but not all kids do.
Smaller groups and increased focus will take additional resources. This level of effort will take money, time, supplies, training, and additional teachers. But you know what? We need to invest properly in our future, even if it means other things suffer. What other things? I don't know ... like a sports arena? Check out the clip below from All In With Chris Hayes which discusses "how a city closing schools at least partly due to money is willing to invest $100 million in building a basketball arena for a private university."
Grouping is also something that needs to be done very carefully. Race and class need to be considered so that the groups are not entirely homogeneous. The groups should be very fluid, allowing kids to pass between in a structured manner. The idea is not to completely separate students of differing ability from one another, but to augment the original lessons with deeper experiences for some groups and basic building for the others. The lesson planning should be structured so that the smaller groups should interact very frequently, since most of the basic content taught should be the same. The objective should always be to move students upward, not to lock them into these groups so that they can't advance no matter what they do (a failure of many of the Gifted and Talented, Honors, and even AP courses that I've seen).
Is this impossible? Nope. Difficult? Definitely. And it's a long term process. But it's well worth the effort.