my respite providers as a kid were my grandparents. They would set me up to spend the week or weekend with my grandparents and take the rest of the family (3 other kids) somewhere without me. I actually liked it because it was always peaceful and predictable there.
Respite providers are supposed to provide comfort and safety, and maybe the first few times could be uncomfortable because it is all new, the surroundings, the routine, etc.
But with experience I would expect it to be OK as I learned the routine, became familiar with the provider, smells, sounds, and all the other stuff that was strange to me.
If it is possible, maybe you could go to the site or meet up with the individual or individuals who will be providing the care. Getting information ahead of time can help prepare for the changes involved.
That may make it easier when it is time to stay there.