I only know about court ordered return to custody.
Usually it's you have to prove you are for so many months 1) drug/alchohol free, 2) have permanent housing, 3) gainfully employed, or 4) some sort of income coming in, 5) child care arrangement.
Do you have an attorney? All this should have been outlined before any paper work was agreed.
If you don't have an attorney, see if you can get a court appointed one. What you don't want is your brother building a case for terminating your parental rights. It's really easy to bag on a homeless, jobless mother who has other issues. Is 3 months considered "not trying hard enough"? Is a year? If nothing is in writing, talk is cheap and doesn't hold up in court.
Just make sure your rights are protected.
As for out of family adoption, and never seeing the child again. Closed adoptions almost never happen any more, unless it's an overseas adoption. Even kids being adopted from foster care have provisions written in that they can see siblings, relatives and even parents. You can have sh***y family members, and the child might do better placed outside of the family, with lots of visitation for kin. Ever state is a bit different, since adoption laws are not uniform everywhere.
Hunt around for a free legal aid clinic as ask if everything is on the up and up. Peace of mind is priceless.
Hope things turn around for you soon.