There are very good developers, that can create great products out of nowhere without thinking too much. Well, I'm not one of those, it takes me a while to get the right idea, and several times I had to throw everything I've bee working on and start from scratch.
But even in situations like these, there are some things you learn from time to time.
This is something I learned from Robert Martin's Clean Code (but I think it just takes common sense). When your components are not meaningful and don't relate to each other in an almost 'natural' way, you need to refactor it. Your system might work beautifully, but your internal structure is a mess and it will create problems to future maintainers.
Keep it simple
Complex design leads to complex code, complex code leads to complex tests, and the mess has now started. The design has to be as simplest as possible, and your code has to be consequent to this. And by code I mean production code and the test units related, because that it's maintainable code too!.
Check KISS Principle on Wikipedia.
System components and inner objects must follow the natural law, die. While shorter your objects live the better, are easier to handle, to test, and to refactor. Including memory usage improvements, as your objects are created and dropped as soon as they are used.
This is something I find very useful, and I discovered by playing around with Google's Guice project.
Check the patterns once a while, review your architecture, don't reinvent the wheel. There are several good references for it, but I tend to enjoy Martin Fowler's patterns (and the articles describing them).
Have a look at the LMAX architecture.