Not able to replicate what a browser (javascript disabled) is doing

[bruce via curl-users]

Hi.

Trying to test out a couple of sites that implement javascript. The
test case for the browser - Opera is to run the browser/target site
with javascript disabled. In this case the browser returns partial
content.

I copied/modified the cmd used by the browser using the "copy as Curl"
function and have the following curl that I'm testing. However, I'm
getting a situation where the process timesout, and doesn't return any
data.

Using " https://mymobile.icc.edu/app/catalog/classSearch/" as the
target url works in different test browsers.

The test curl follows:

curl -v -A "Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
Firefox/38.0" --cookie-jar a.lwp --cookie a.lwp -H
'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language:
en-US,en;q=0.5' -H 'Connection: keep-alive' -H 'Host:
mymobile.icc.edu' -H 'Upgrade-Insecure-Requests: 1' -L
'https://mymobile.icc.edu/app/catalog/classSearch/'

At this point, I'm curious as to why the process hangs but the browser
(with javascript disabled) doesn't.

Any thoughts/comments??

thanks


ps. If I can resolve this I've got another diff/sim issue with another
test site as well!

thanks
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.ha

[Ray Satiro via curl-users]

Post by bruce via curl-users
Trying to test out a couple of sites that implement javascript. The
test case for the browser - Opera is to run the browser/target site
with javascript disabled. In this case the browser returns partial
content.
I copied/modified the cmd used by the browser using the "copy as Curl"
function and have the following curl that I'm testing. However, I'm
getting a situation where the process timesout, and doesn't return any
data.
Using " https://mymobile.icc.edu/app/catalog/classSearch/" as the
target url works in different test browsers.
curl -v -A "Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
Firefox/38.0" --cookie-jar a.lwp --cookie a.lwp -H
mymobile.icc.edu' -H 'Upgrade-Insecure-Requests: 1' -L
'https://mymobile.icc.edu/app/catalog/classSearch/'
At this point, I'm curious as to why the process hangs but the browser
(with javascript disabled) doesn't.

I can't reproduce this. I went as far back as 7.39.0 and tried a number
of ssl backends including openssl, winssl, wolfssl and mbedtls. If you
can still reproduce this please give your curl -V and the verbose output
from the command so we can see where it hangs. Also, your use of the
Accept-Encoding header will, if the response is compressed, prevent curl
from automatically decompressing the response using the supported
decompression methods. I suggest use --compressed [1] instead.

[1]: https://curl.haxx.se/docs/manpage.html#--compressed

[bruce via curl-users]

Aha!

Hi Ray. Thanks for your input. Using the following, I was able to get
output similar to what the browser generated.

Now my issue is how to get the "Career" values. The target url in the
browser with javascript lists "Undergrad/Graduate" as the
select/option list. In inspecting the FF/Debug data for the
page/javascript I can't figure out where this is derived from. I can
easily see the terms/subjects from the api calls. The career data
(undergrad/graduate) isn't happening for me though!

curl -v -A "Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
Firefox/38.0" --cookie-jar a.lwp --cookie a.lwp --compressed -H
'Connection: keep-alive' -H 'Host: mymobile.icc.edu' -H
'Upgrade-Insecure-Requests: 1' -L
'https://mymobile.icc.edu/app/catalog/classSearch/'


Thoughts/comments???

thanks!


On Tue, Dec 19, 2017 at 3:09 PM, Ray Satiro via curl-users

Post by bruce via curl-users
Trying to test out a couple of sites that implement javascript. The
test case for the browser - Opera is to run the browser/target site
with javascript disabled. In this case the browser returns partial
content.
I copied/modified the cmd used by the browser using the "copy as Curl"
function and have the following curl that I'm testing. However, I'm
getting a situation where the process timesout, and doesn't return any
data.
Using " https://mymobile.icc.edu/app/catalog/classSearch/" as the
target url works in different test browsers.
curl -v -A "Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
Firefox/38.0" --cookie-jar a.lwp --cookie a.lwp -H
mymobile.icc.edu' -H 'Upgrade-Insecure-Requests: 1' -L
'https://mymobile.icc.edu/app/catalog/classSearch/'
At this point, I'm curious as to why the process hangs but the browser
(with javascript disabled) doesn't.
I can't reproduce this. I went as far back as 7.39.0 and tried a number of
ssl backends including openssl, winssl, wolfssl and mbedtls. If you can
still reproduce this please give your curl -V and the verbose output from
the command so we can see where it hangs. Also, your use of the
Accept-Encoding header will, if the response is compressed, prevent curl
from automatically decompressing the response using the supported
decompression methods. I suggest use --compressed [1] instead.
[1]: https://curl.haxx.se/docs/manpage.html#--compressed
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html

-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users

[Ray Satiro via curl-users]

Post by bruce via curl-users
Now my issue is how to get the "Career" values. The target url in the
browser with javascript lists "Undergrad/Graduate" as the
select/option list. In inspecting the FF/Debug data for the
page/javascript I can't figure out where this is derived from. I can
easily see the terms/subjects from the api calls. The career data
(undergrad/graduate) isn't happening for me though!

That is outside the scope of the list. I did look at it though and
couldn't figure out what you mean.

Post by bruce via curl-users
curl -v -A "Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
Firefox/38.0" --cookie-jar a.lwp --cookie a.lwp --compressed -H
'Connection: keep-alive' -H 'Host: mymobile.icc.edu' -H
'Upgrade-Insecure-Requests: 1' -L
'https://mymobile.icc.edu/app/catalog/classSearch/'

I would lose the Host header as well. Headers stay the same on redirects
[1] and you may be redirected to a different host so you wouldn't want
to send the wrong host in that next transfer. The connection header is
also unnecessary.

[1]: https://curl.haxx.se/docs/manpage.html#-H



-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: