About unicast TCP/IP
When initiating a unicast TCP/IP connection, the destination has to be known in advance, it consists of an IP address and a port. Your local connecting tool chooses a local IP address (interface) plus a local port and binds to it; it may ask the OS to pick the right address (from which the destination can be reached) and a port (ephemeral port). One way or another there will be some bound local IP address and port, and some destination IP address and port.
The tuple ( local_IP, local_port, destination_IP, destination_port ) identifies the connection. Only responses that come to the right local IP and the right local port from the right destination IP and the right destination port will be recognized by the OS as belonging to the connection and thus get to your tool. If any response comes in from another IP address then it will not get to the tool*.
* ICMP is an exception. E.g. when your tool tries to establish a TCP/IP connection, an intermediate network node may respond with some kind of "destination unreachable" message and this will be recognized as associated with the connection, the tool will be informed. I think your question is not about ICMP responses, it is about the situation when you get an actual response to your HTTP request.
What is the address then?
For HTTP to work, the transport layer connection has to be established first, this means TCP. And if the TCP/IP connection has been established then there is only one IP address that can talk to your tool: the address in the tuple, i.e. the destination address the tool itself had requested.
If you tell your tool to explicitly connect to an IP address then this is the address. If you tell your tool to connect to a URL then the most straightforward way to know the IP address is to let the tool tell you (after it resolves the URL). wget does it by default; curl does it with --verbose.
Alleged complications
Note if there is a redirection on HTTP level (e.g. "301 Moved Permanently") then it will not happen automatically and transparently inside the same TCP/IP connection. If your tool chooses to follow the redirection then it has to start a new TCP/IP connection to the new destination. The situation with the new connection is exactly like it was with the old connection; everything I wrote applies to the new connection as well.
If the destination is a proxy that connects to another server then a response from the server will get to the proxy and only then the proxy will send it to you within the TCP/IP connection your tool has established with the proxy. The responses will get to your tool from the proxy. The proxy may or may not add information about the actual server (inside HTTP header), but the IP address your tool sends to and receives from will be the one of the proxy all the time.
The assumption (from your comments) that "the web server would be routed through a proxy with incoming requests, but not outgoing requests" is false.
Summary
It's all about how unicast TCP/IP works, in principle this is very simple: your connecting tool uses some destination IP address in order to connect, then the address is the only address from which data will be assigned to the connection and delivered to the tool.
The conclusion is: there is no point in trying to find out what IP address a response came from. The sole fact a HTTP response has been delivered to your tool proves it came from the IP address the tool had connected to. This address was known to the tool even before the connection was started, get it from the tool.