Sure, sudo is a setuid binary, but it’s a fairly simple program, and at some point, you have to trust the code.
Have to trust the code ? doas for OpenBSD was created because of issues with sudo.
Talking with deraadt and millert, however, I wasn’t quite alone. There were some concerns that sudo was too big, running too much code in a privileged process. And there was also pressure to enable even more options, because the feature set shipped in base wasn’t big enough.