they don’t even know they’re trying to guess words in the first place.
That is true, but the math is still the same regardless.
Suppose you had a word list of 1,000 five letter words. Each of your passphrases is 5 words long. That means you have 1,000^5 possible combinations of passwords, which is an entropy of ~49.8 bits. Even though each passphrase is going to be 29 characters long (5 five letter words plus 4 spaces in between), the password wasn’t generated character by character.
By contrast, suppose you used all 95 characters on the (US) keyboard, an 8 character password has 95^8 combinations, which is an entropy of ~52.6 bits. Even though the passphrase has 21 more characters than the password, the password still has more entropy.
Big grain of salt here: You can get a huge word list and remember much longer passphrases easily, but the point is to show that the number of characters doesn’t dictate the security of a password. If someone were to brute force a passphrase character-by-character, it would hold up very well, but a) Not many people use passphrases and b) It’s far more common to use password dictionaries than to brute force.
P.S. If someone found your word list, they could probabilistically brute force your passwords. For example, if 75% of your five letter words started with the letter S, they could deduce that most of the words likely start with S, and they’ve already eliminated a few characters to brute force.