I wrestle with this question every time I put out a new version of my site, or anything else I make, as mentioned in my post about The Itch. When The Itch appears, you want to take all that you learned in just the last couple weeks and just do the whole project over again.
Not so fast bucko.
Reasons for refactoring are everywhere, far and wide. But whether you should comes down to a question of Technical Debt. Compare: The project as it is now, with all it's quirks and gotchas that you know well - or a completely new code base, with new quirks to learn, but easier to manage? Which is better?
No two projects are alike in this question either. Refactoring might make the developer experience an incredible, state-of-the-art majesty of innovation and efficiency, but it might come at the cost of a bunch more libraries to load and render time on slower machines, etc. Maybe you're refactoring to get rid of a bunch of libraries and resources, at the cost of learning how to do tasks without those resources.
Just some stuff to consider.