Fix compiler warnings with open_to_operation.c

Fix compiler warnings with open_to_operation.c

Other sidenotes:
- Remove unused variables with main(..)
- Convert errx/exit with -1 to errx/exit with 1
- Fix a bogus test in try_directory_open
  (expected_errno == expected_errno -> errno == expected_errno) [*]
- Fix some warnings related to discarded qualifiers
- Remove a bogus else-statement at the end of check_mmap_exec(..) in the
  successful case. mmap(2), POSIX, Linux, etc all don't state what the
  behavior is when mixing O_WRONLY + PROT_EXEC, so assume success for now to
  get the test program to pass again.

PR: 201286 [*]
MFC after: 1 week
Submitted by: David Binderman <dcb314@hotmail.com>
Sponsored by: EMC / Isilon Storage Division