Were you raised by one family member?
I have heard it is harder for women who have been raised with both a mother and a father figure to parent a child single than it is for one who has been raised with a single parent figure.
You mention your mom and your "real mom", so that's why I ask.
My thoughts on TTC, after a long bout of problems is this.
The focus of getting pregnant is to have a family.
I know a lot of people want an infant, but they don't examine WHY they want an infant. Many of them want an infant thinking that the child would possibly never know they were adopted.
They will know. Kids are observant. Myself, hubby and I are going to adopt an older child...anywhere from 6 to 10 years of age, because we know there is a child out there, right now, needing a mommy and daddy, and because it is harder for them to find a mommy and daddy, we're going older.
Think of the stuff, which is wonderful if you have one of your own, that we don't "have to worry about" when adopting an older child.
No diapers, instantly being able to know from the child themself what hurts, not worrying about SIDS, etc etc.
Of course, if I had a baby naturally, I would welcome those challenges, but if I'm not having one naturally, why go through it?
You can ask most parents, its when kids hit about 4 or 5 that the fun begins.
That's just my .02
Best of luck with whatever you decide.
