I don't think we'll be worse. Carroll likely won't be any worse than Carr - who by all accounts was terrible 50% of the time. Claiborne was injury prone and had his best season last year, and even that was just a slightly better than average season.
I honestly think if Brown continues to develop, paired with Scandrink and hopefully a player or two in a very deep secondary draft, the back end will in fact be better. Should we be worse, however, I doubt it'll be a significant drop off.
I could be wrong. All I know is that every time a QB needed to carve this secondary up with the game on the line, 9 times out of 10 they've been able to do just that the past 4-5 years, if not longer. We've had a history of making Colt McCoys and other average journeyman look like studs. Keeping the status quo wasn't the answer.