Discussion:
How to intercept curl to extract the raw requests and the raw responses?
(too old to reply)
Peng Yu
2018-02-09 15:30:01 UTC
Permalink
Hi,

In the following example, `curl` should send an HTTP GET request to httpbin.org.

$ curl -g -sS http://httpbin.org/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "curl/7.57.0"
},
"origin": "165.91.87.88",
"url": "http://httpbin.org/get"
}

The request probably starts with the following lines and I'd like to
see them in the raw request. Is there a way to intercept curl so that
I can see the raw requests as well as the raw responses?

GET /get HTTP/1.1
Host:httpbin.org

PS. I tried the following HTTP proxy, but it can not show the raw request.

https://github.com/abhinavsingh/proxy.py
--
Regards,
Peng
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/m
Ray Satiro
2018-02-09 22:01:34 UTC
Permalink
Post by Peng Yu
In the following example, `curl` should send an HTTP GET request to httpbin.org.
$ curl -g -sS http://httpbin.org/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "curl/7.57.0"
},
"origin": "165.91.87.88",
"url": "http://httpbin.org/get"
}
The request probably starts with the following lines and I'd like to
see them in the raw request. Is there a way to intercept curl so that
I can see the raw requests as well as the raw responses?
GET /get HTTP/1.1
Host:httpbin.org
It depends, it's subject to some breakage. If it's just for your eyes
and you're not feeding it to another program you could parse it out of
verbose or trace-ascii (verbose is better because it doesn't split
headers longer than 64 bytes like trace-ascii does). Sent headers start
with > and received headers start with <

curl -v -fsS google.com 2>&1 1>/dev/null | grep -E "^(<|>|curl: )"

(in windows use NUL instead of /dev/null)

or as a function in bash

headers(){ curl -v -fsS "$1" 2>&1 1>/dev/null | grep -E "^(<|>|curl: )"; }
headers google.com

(the curl: is to show errors, remove that part if you don't want to see
them)

Any library used by curl could output to stderr its own > and < at the
same time which would get caught up, or curl could change it. If you
want parseable defined sent header lines you would have to use libcurl's
CURLOPT_DEBUGFUNCTION [1][2] and handle CURLINFO_HEADER_OUT.


[1]: https://curl.haxx.se/libcurl/c/CURLOPT_DEBUGFUNCTION.html
[2]:
https://github.com/curl/curl/blob/curl-7_58_0/docs/examples/debug.c#L100
Ray Satiro
2018-02-09 22:05:05 UTC
Permalink
Post by Ray Satiro
Post by Peng Yu
In the following example, `curl` should send an HTTP GET request to httpbin.org.
$ curl -g -sS http://httpbin.org/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "curl/7.57.0"
},
"origin": "165.91.87.88",
"url": "http://httpbin.org/get"
}
The request probably starts with the following lines and I'd like to
see them in the raw request. Is there a way to intercept curl so that
I can see the raw requests as well as the raw responses?
GET /get HTTP/1.1
Host:httpbin.org
It depends, it's subject to some breakage. If it's just for your eyes
and you're not feeding it to another program you could parse it out of
verbose or trace-ascii (verbose is better because it doesn't split
headers longer than 64 bytes like trace-ascii does). Sent headers
start with > and received headers start with <
curl -v -fsS google.com 2>&1 1>/dev/null | grep -E "^(<|>|curl: )"
(in windows use NUL instead of /dev/null)
or as a function in bash
headers(){ curl -v -fsS "$1" 2>&1 1>/dev/null | grep -E "^(<|>|curl: )"; }
headers google.com
(the curl: is to show errors, remove that part if you don't want to
see them)
Any library used by curl could output to stderr its own > and < at the
same time which would get caught up, or curl could change it. If you
want parseable defined sent header lines you would have to use
libcurl's CURLOPT_DEBUGFUNCTION [1][2] and handle CURLINFO_HEADER_OUT.
[1]: https://curl.haxx.se/libcurl/c/CURLOPT_DEBUGFUNCTION.html
https://github.com/curl/curl/blob/curl-7_58_0/docs/examples/debug.c#L100
 -f pre-empts 4xx response headers with an error. So remove -f if you
want to see 4xx response headers
Keefe Tang
2018-02-23 04:32:01 UTC
Permalink
Yes. You can add the verbose flag -v in your command to see the
request and response to and from the server.
Post by Peng Yu
Hi,
In the following example, `curl` should send an HTTP GET request to httpbin.org.
$ curl -g -sS http://httpbin.org/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "curl/7.57.0"
},
"origin": "165.91.87.88",
"url": "http://httpbin.org/get"
}
The request probably starts with the following lines and I'd like to
see them in the raw request. Is there a way to intercept curl so that
I can see the raw requests as well as the raw responses?
GET /get HTTP/1.1
Host:httpbin.org
PS. I tried the following HTTP proxy, but it can not show the raw request.
https://github.com/abhinavsingh/proxy.py
--
Regards,
Peng
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://
James Stephenson
2018-02-23 15:07:21 UTC
Permalink
You can create a trace file when you issue the curl command. Here is a
simple batch file I am testing to do an HTTP GET. At the end of it is the
reference to dump the session into a Trace file.

@Echo Off
REM Submission URL = %1
REM Return File = %2
REM Trace File = %3
REM User ID:Password = %4
REM Function = %5 (GET)

m:\UP\BFHD\Curl\curl.exe --user %4 --cacert cacert.pem -H "Content-Type:
application/json; charset=utf-8" -H "SOAPAction:" --tlsv1.2 -X %5 %1 -o %2
-v --trace-ascii %3
Post by Keefe Tang
Yes. You can add the verbose flag -v in your command to see the
request and response to and from the server.
Post by Peng Yu
Hi,
In the following example, `curl` should send an HTTP GET request to
httpbin.org.
Post by Peng Yu
$ curl -g -sS http://httpbin.org/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "curl/7.57.0"
},
"origin": "165.91.87.88",
"url": "http://httpbin.org/get"
}
The request probably starts with the following lines and I'd like to
see them in the raw request. Is there a way to intercept curl so that
I can see the raw requests as well as the raw responses?
GET /get HTTP/1.1
Host:httpbin.org
PS. I tried the following HTTP proxy, but it can not show the raw
request.
Post by Peng Yu
https://github.com/abhinavsingh/proxy.py
--
Regards,
Peng
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Loading...