How to intercept curl to extract the raw requests and the raw responses?

[Peng Yu]

Hi,

In the following example, `curl` should send an HTTP GET request to httpbin.org.

$ curl -g -sS http://httpbin.org/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "curl/7.57.0"
},
"origin": "165.91.87.88",
"url": "http://httpbin.org/get"
}

The request probably starts with the following lines and I'd like to
see them in the raw request. Is there a way to intercept curl so that
I can see the raw requests as well as the raw responses?

GET /get HTTP/1.1
Host:httpbin.org

PS. I tried the following HTTP proxy, but it can not show the raw request.

https://github.com/abhinavsingh/proxy.py

[Ray Satiro]

Post by Peng Yu
In the following example, `curl` should send an HTTP GET request to httpbin.org.
$ curl -g -sS http://httpbin.org/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "curl/7.57.0"
},
"origin": "165.91.87.88",
"url": "http://httpbin.org/get"
}
The request probably starts with the following lines and I'd like to
see them in the raw request. Is there a way to intercept curl so that
I can see the raw requests as well as the raw responses?
GET /get HTTP/1.1
Host:httpbin.org

It depends, it's subject to some breakage. If it's just for your eyes
and you're not feeding it to another program you could parse it out of
verbose or trace-ascii (verbose is better because it doesn't split
headers longer than 64 bytes like trace-ascii does). Sent headers start
with > and received headers start with <

curl -v -fsS google.com 2>&1 1>/dev/null | grep -E "^(<|>|curl: )"

(in windows use NUL instead of /dev/null)

or as a function in bash

headers(){ curl -v -fsS "$1" 2>&1 1>/dev/null | grep -E "^(<|>|curl: )"; }
headers google.com

(the curl: is to show errors, remove that part if you don't want to see
them)

Any library used by curl could output to stderr its own > and < at the
same time which would get caught up, or curl could change it. If you
want parseable defined sent header lines you would have to use libcurl's
CURLOPT_DEBUGFUNCTION [1][2] and handle CURLINFO_HEADER_OUT.


[1]: https://curl.haxx.se/libcurl/c/CURLOPT_DEBUGFUNCTION.html
[2]:
https://github.com/curl/curl/blob/curl-7_58_0/docs/examples/debug.c#L100

[Ray Satiro]

Post by Ray Satiro

Post by Peng Yu
In the following example, `curl` should send an HTTP GET request to httpbin.org.
$ curl -g -sS http://httpbin.org/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "curl/7.57.0"
},
"origin": "165.91.87.88",
"url": "http://httpbin.org/get"
}
The request probably starts with the following lines and I'd like to
see them in the raw request. Is there a way to intercept curl so that
I can see the raw requests as well as the raw responses?
GET /get HTTP/1.1
Host:httpbin.org

It depends, it's subject to some breakage. If it's just for your eyes
and you're not feeding it to another program you could parse it out of
verbose or trace-ascii (verbose is better because it doesn't split
headers longer than 64 bytes like trace-ascii does). Sent headers
start with > and received headers start with <
curl -v -fsS google.com 2>&1 1>/dev/null | grep -E "^(<|>|curl: )"
(in windows use NUL instead of /dev/null)
or as a function in bash
headers(){ curl -v -fsS "$1" 2>&1 1>/dev/null | grep -E "^(<|>|curl: )"; }
headers google.com
(the curl: is to show errors, remove that part if you don't want to
see them)
Any library used by curl could output to stderr its own > and < at the
same time which would get caught up, or curl could change it. If you
want parseable defined sent header lines you would have to use
libcurl's CURLOPT_DEBUGFUNCTION [1][2] and handle CURLINFO_HEADER_OUT.
[1]: https://curl.haxx.se/libcurl/c/CURLOPT_DEBUGFUNCTION.html
https://github.com/curl/curl/blob/curl-7_58_0/docs/examples/debug.c#L100

 -f pre-empts 4xx response headers with an error. So remove -f if you
want to see 4xx response headers

[Keefe Tang]

Yes. You can add the verbose flag -v in your command to see the
request and response to and from the server.

Post by Peng Yu
Hi,
In the following example, `curl` should send an HTTP GET request to httpbin.org.
$ curl -g -sS http://httpbin.org/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "curl/7.57.0"
},
"origin": "165.91.87.88",
"url": "http://httpbin.org/get"
}
The request probably starts with the following lines and I'd like to
see them in the raw request. Is there a way to intercept curl so that
I can see the raw requests as well as the raw responses?
GET /get HTTP/1.1
Host:httpbin.org
PS. I tried the following HTTP proxy, but it can not show the raw request.
https://github.com/abhinavsingh/proxy.py
--
Regards,
Peng
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html

-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://

[James Stephenson]

You can create a trace file when you issue the curl command. Here is a
simple batch file I am testing to do an HTTP GET. At the end of it is the
reference to dump the session into a Trace file.

@Echo Off
REM Submission URL = %1
REM Return File = %2
REM Trace File = %3
REM User ID:Password = %4
REM Function = %5 (GET)

m:\UP\BFHD\Curl\curl.exe --user %4 --cacert cacert.pem -H "Content-Type:
application/json; charset=utf-8" -H "SOAPAction:" --tlsv1.2 -X %5 %1 -o %2
-v --trace-ascii %3

Post by Keefe Tang
Yes. You can add the verbose flag -v in your command to see the
request and response to and from the server.

Post by Peng Yu
Hi,
In the following example, `curl` should send an HTTP GET request to

httpbin.org.

Post by Peng Yu
$ curl -g -sS http://httpbin.org/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "curl/7.57.0"
},
"origin": "165.91.87.88",
"url": "http://httpbin.org/get"
}
The request probably starts with the following lines and I'd like to
see them in the raw request. Is there a way to intercept curl so that
I can see the raw requests as well as the raw responses?
GET /get HTTP/1.1
Host:httpbin.org
PS. I tried the following HTTP proxy, but it can not show the raw

request.

Post by Peng Yu
https://github.com/abhinavsingh/proxy.py
--
Regards,
Peng
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html

-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html