Do you need a recording device with a microphone, or do you already have a microphone which you want to plug in? One option for mic and recorder in one device is a Zoom H1 (or ‘Zoom Handy H1’) which can be found for around $100 retail (on Amazon, for example). It has a built-in stereo microphone and runs off one AA battery. It also has a 1/8" input socket in case you want to plug in an external sound source.
It records in stereo or mono on microSD and it’s got a lot of features - that’s pretty much unavoidable with Zoom gadgets - but most of the time you can simply get away with pressing the red button at the center, it’s as easy as with an old cassette recorder.
There are various quality options, and most of them are good enough for CD quality.
If you move the microSD to a PC you’ll find that each recording is stored as a .wav file (unless configured to record as .mp3). Or you can just connect the device directly to a PC with a USB cable instead, as with a digital camera.
Disclaimer: I have not used the H1 myself, but it is in the same range of products as my more complicated H4 model, with similar microphones etc. The sound quality is good. My H4 came with some studio software for the PC as well, Cube something, I don’t see that in the ads for the H1. But I never used that software anyway. For processing (e.g. creating a CD) you can use just about anything which works for you.