So I feel like you either want to ride bikes for the sake of riding bikes, or you want to ride bikes to get better. I think tons of people just ride to ride. They are happy with just pedaling along and not pushing themselves towards progressing further. I think about the people who go on the tow path and ride for an hour and they are just wanting to be out there, riding. They don't look at the time, or their speed, they just don't care about that stuff. Some days I like to be that guy too. That's totally fine to do forever, but you have to be okay with that if that's what you're doing.
If you want to get better, you can. I don't believe that you hit a point right now where you can't grow more as a cyclist. Every year I grow, even if it's hard to see sometimes. But getting faster doesn't come easy, you have to work really really hard to get where you want to be. Some times even that isn't enough.
“It never gets easier, you just go faster.” -Greg LeMond
I remind myself of this all the time. I ride and I think, "shit, this was so much easier last year. Did I get that slow?" Then I go home and upload my file and realize I'm actually riding faster, it's just become the new normal. You can do that same effort you thought was so hard, and it's your new tempo.
Bottom line is that if you set your mind to something, you can do it. You just need to want it bad enough.