This probably only meets some of your criteria, but there are lots of free audiobooks available through librivox at librivox.org. They make free public domain audiobooks by volunteer effort; reader quality varies widely (there are some great readers, and some not very good ones), and the fact that the books have to be public domain limits the range of content.
Here's how it fits your criteria:
"--mp3 format, or something similar that can be played on a digital media player"
Yes.
"--Inexpensive, less than $20/month"
Yes. Free.
"--Ability to check out about ten books per month at least; preferably, a flat fee for unlimited audiobooks."
Yes. Unlimited.
"--A large selection of nonfiction including fairly advanced topics, especially science. Probably only possible for services with a very large selection, period. I'm looking for something around fifty to a hundred thousand books, total, which will probably include two or three hundred I'd be interested in."
There is some nonfiction, but due to the public domain constraint, it's mostly old, so not good for things where being out-of-date matters, which is most of science. (You can, however, listen to things like original works by Charles Darwin (and probably more old scientific works but it would take some effort to find them.))
_________________
Now convinced that I'm a bit autistic, but still unsure if I'd qualify for a diagnosis, since it causes me few problems. Apparently people who are familiar with the autism spectrum can readily spot that I'm a bit autistic, though.