In Unix/Linux all the communication to the terminal (terminal emulator), i.e. to the device (or program) that displays text for you to see, gets to it within a single stream. Printable characters are within the stream along with characters that move the cursor (e.g. backspace, newline) and escape sequences that can clear the screen, set colors, move the cursor in more complex way or do other things.
When you save the output of program(s) that believe they are sending to a terminal, you save all this.
watch is designed to print to a terminal and it does not change its behavior even when printing to a pipe or to a regular file. Other programs may change their behavior (example). script is designed to intercept output by creating a tty the child and its descendants will use and they will believe they are printing to a terminal, while they are printing to script which acts as a proxy between them and the actual tty. (This is why script can be used to capture from programs that change their behavior, if this is what you want.)
There is no side channel that could carry unprintable characters and escape sequences separately and thus allow you to save only the "main channel" easily. There is just one channel. In some cases the terminal even responds and it does it as if you typed because it's the only channel it has. (Example: Why is the output of Device Status Report sent to the standard input instead of output?)
The "something weird" you saw in your file is normal. Usually we don't see it as weird because the terminal interprets all the "weirdness". In fact the "weirdness" is there exactly for the terminal to interpret it.
There are ways to process your file(s) later. See How to clean up output of Linux script command and How to convert escape sequences to text while preserving display format.
Use script with --log-timing, then you can use scriptreplay to play it back. Example:
script --log-timing script.tm --log-out script.out --command 'timeout 10 watch date'scriptreplay --log-timing script.tm --log-out script.out --maxdelay 0.8