Then you know I have them, but you still don't know if I sent them back there, or some other party did.
You are thinking that, since it's easy to list all the possible chess moves in the next move, that you can tell for sure how the board got in its current state after 100 moves.