As with any programming task, it is easier to handle if you focus on the common case. You can use clever speedup tricks, make cleaner code or any other thing you want if you do that.
The common case is that people do not go deep at all. This of course does not prove that there is nothing sinister going on with search engines, but some other "proof" is needed.
I think it is common knowledge that the top results are quite often in part paid for or promoted in some other way. That can be troubling in itself of course, ie when lies are pushed
