Memory increase with https requests

[madana gopal]

Hi Team,

We are using curl version 7.35 and 7.40 in our project and running in
linux. When we make https requests via *curl_easy_perform()* call, there is
a regular memory increase of process VIRT and RAM sections seen from top
command. Amount of memory increase is based on number of https requests
made. *curl_easy_cleanup() *call is made at the end of every request.

Memory increase is not seen with http requests.

Please let me know, do we have any issues related to this already and there
is some standard way to make curl calls for https requests.If there is a
known patch for this, please share the same. We may not want to upgrade the
curl package at this moment, but can apply the patch.

Thanks.

Regards,
Madan

[Daniel Stenberg]

We are using curl version 7.35 and 7.40 in our project and running in linux.
When we make https requests via *curl_easy_perform()* call, there is a
regular memory increase of process VIRT and RAM sections seen from top
command. Amount of memory increase is based on number of https requests
made. *curl_easy_cleanup() *call is made at the end of every request.
Memory increase is not seen with http requests.
Please let me know, do we have any issues related to this already and there
is some standard way to make curl calls for https requests.If there is a
known patch for this, please share the same. We may not want to upgrade the
curl package at this moment, but can apply the patch.

It is not surprising that HTTPS uses (much) more memory than plain HTTP. The
added TLS layer requires a fair amount of memory.

You're using fairly old versions of libcurl and we've fixed many bugs since
then, but I can't recall any significant memory leak in any of those version
that would make the memory use continue to grow unbounded. The memory use you
see is most likely perfectly fine, expected and will all be freed when your
application exits. You can of course use memory debugging tools such as
valgrind to verify this if you want to.

Finally: you seem to be using libcurl, so you might want to consider the
curl-library mailing list if you want to dive into libcurl details.

[madana gopal]

Hi Daniel,

Thanks for the details. We have a requirement that the app won't want to
get exited, but want to get the memory freed on need. Do we have any way to
handle this,?please share can we achieve this in any way?.

For libcurl related requests, I will speak in curl-library.

Thanks.

Regards,
Madan

Post by madana gopal
We are using curl version 7.35 and 7.40 in our project and running in

Post by madana gopal
linux. When we make https requests via *curl_easy_perform()* call, there is
a regular memory increase of process VIRT and RAM sections seen from top
command. Amount of memory increase is based on number of https requests
made. *curl_easy_cleanup() *call is made at the end of every request.
Memory increase is not seen with http requests.
Please let me know, do we have any issues related to this already and
there is some standard way to make curl calls for https requests.If there
is a known patch for this, please share the same. We may not want to
upgrade the curl package at this moment, but can apply the patch.

It is not surprising that HTTPS uses (much) more memory than plain HTTP.
The added TLS layer requires a fair amount of memory.
You're using fairly old versions of libcurl and we've fixed many bugs
since then, but I can't recall any significant memory leak in any of those
version that would make the memory use continue to grow unbounded. The
memory use you see is most likely perfectly fine, expected and will all be
freed when your application exits. You can of course use memory debugging
tools such as valgrind to verify this if you want to.
Finally: you seem to be using libcurl, so you might want to consider the
curl-library mailing list if you want to dive into libcurl details.
--
/ daniel.haxx.se
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html

[Daniel Stenberg]

We have a requirement that the app won't want to get exited, but want to get
the memory freed on need. Do we have any way to handle this?

No.

libcurl will only use as much memory as it thinks it needs and there's no way
to poke it and tell it to use less memory all of a sudden. You can possibly
tweak parameters etc to make it cache less data to somewhat reduce its memory
footprint, but I don't think you can shave off much that way. The TLS
libraries tend to consume way more memory than the rest of libcurl does.

The way libc usually works, it will grow the heap for the application
depending on need during its lifetime but it will not shrink it. Thus you
won't get the heap back "for real" until the program exits and the peak memory
amount will thus decide how large heap your application will have.

[madana gopal]

ok, thanks Daniel.

Regards,
Madan

We have a requirement that the app won't want to get exited, but want to

get the memory freed on need. Do we have any way to handle this?

No.
libcurl will only use as much memory as it thinks it needs and there's no
way to poke it and tell it to use less memory all of a sudden. You can
possibly tweak parameters etc to make it cache less data to somewhat reduce
its memory footprint, but I don't think you can shave off much that way.
The TLS libraries tend to consume way more memory than the rest of libcurl
does.
The way libc usually works, it will grow the heap for the application
depending on need during its lifetime but it will not shrink it. Thus you
won't get the heap back "for real" until the program exits and the peak
memory amount will thus decide how large heap your application will have.
--
/ daniel.haxx.se
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html