It may be harder than you expect to establish that you're not in a business related to your former employer, at least to the extent that you can quickly end a legal action; all I'm saying is, that legal safeguard may not be as helpful as you'd hope.
Particularly if the employer is someone like Microsoft or Google that does, well... a little of everything. As I said, it's definitely worth consulting a lawyer before taking anything for granted.
I'm pretty sure Oracle is a good example of a company that makes you agree that they own everything you think of while employed by them.
A friend of mine became an Oracle employee when the company he worked for got acquired, and ended up needing to get approval from a long way up the org chart to be able to continue doing his phd, since the university had existing claims on ip in his phd research.
For quite a long time he described himself as " technically not being allowed to think anything..."
It's not oracle specificaly it's that if you are say a structural ironworker during the day and make bronze sculptures as a hobby then a judge will understand they are different.
But they might not understand that being a sysadmin is as different to being a web designer in the evening.
As I understand it, Oracle's standard employment agreement includes language saying you agree that Oracle owns every idea you have while employed by them, whether on "paid hours" or not, whether related to databases or software in general or not - if you're a database engineer for Oracle and you invent a better tap washer, Oracle seem to think they'll own your plumbing related invention. There's workarounds for at least limited exceptions to that standard rule, but it's apparently unusual and required signoff at many levels above the typical workers management.
You're absolutely right. There are really two questions here:
* Does my employer own this?
* Will they make me prove that in court?
The damnable thing is that, even if you really do own your work, having to prove that you own it in court may kill your startup by making you run out of money.
The other way of looking at your latter point: I try to work for people who would brag about successful former employees. Enlightened managers cultivate an "alumni network."