Nope, don´t really miss it. I sometimes miss the seating position and the wind protection, although the seat itself was awful for long rides, but I don´t miss the “do it all”. And that, I fear, is mainly because at least in the end, I didn´t really do it all any more. See, I live in Germany. Where it´s not trivial to just ride off road. And I never hat the time or budget to really go on adventures. And while I did the off dirt road on the first two sets of tires, I quickly found myself 99% on regular roads. I was too proud to do full stree tires, so I opted for some 80/20 type ones. Still, gravel was as wild as it got. Don´t get me wrong. I had the AT fpr 10 years. And it was just a great touring bike, even on regular roads. I don´t regret not using it “properly”. Still, that fact made it pretty easy to switch to a different style of bike. Not radically different, since I still have relatively upright seating and good wind protection, but different enough to matter.
So for your situation, I guess look at the way you actually use your current bike. For me, the decision was easy. But if you regularly ride off road, I fully understand your hesitation.
What do you ride then? Because just because you haven an ADV doesn´t have to make you slow. When I ride in Austria, in the alps, with serious switchbacks as well, somehow the big 1200GS are often faster than the sport bikes. Feels like that bumblebee situation, where they actually can´t fly but since nobody told them that, they do it anyway.