I'm non-native, but worked in a factory where English was the main language spoken among people (next to Tagalog... Akane... Cantonese... Sranan..., but I'm rusty in those languages, so I settled for English ).
There was honestly no difference. I managed to get along with a number of people, but in general I remained socially awkward, which expressed itself in some strained relationships with certain co-workers. My English is conversational, and I was able to express many of my thoughts in words. Language was not a problem in that environment.
I think going to a native Anglophone community would be a different beast altogether. At my work, there were different registers/varieties of English being used, even if my Nigerian, Ghanaian, Filipina etc. coworkers were all fluent in English. So you still had to compromise between those variants.
_________________
clarity of thought before rashness of action