Discouraging to read non fiction sometimes

I try to keep a decent balance between fiction and non fiction when I read. The majority is still fiction though.

But sometimes, even if it's a topic I really enjoy, I get so discouraged from reading because I won't remember most of it. What's the main purpose of reading non fiction other than to learn, but how can I learn if I can't remember most of what I'm reading. It feels like such a waste sometimes.