trying to list all message subjects in my inbox using cURL and an imaps url

[Stephen Jarjoura]

Hello;

I only recently learned that cURL supports both SMTP(S) and IMAP(S)
and have some ideas for using it in my scripting. While I've gotten
the SMTPS to work well, I'm having some problems with the IMAPS,
specifically, it echoes back the command and a byte count instead of
returning the requested data.

I'm running cURL on my Win7 desktop.
curl 7.36.0 (x86_64-pc-win32) libcurl/7.36.0 OpenSSL/1.0.1g zlib/1.2.8
WinIDN libssh2/1.4.3

Here are some examples:
This is what I get when I send the command to an Exchange 2010 mail server.

C:\>curl imaps://email.my.domain/Inbox -u ***@my.domain:password
* LIST (\Marked \HasNoChildren) "/" INBOX

From this page: http://curl.haxx.se/mail/lib-2013-03/0104.html

which reports "--url imap://mail.example.com -X "EXAMINE INBOX" -
Performs a message list on the user's inbox"

But here's what I get:
C:\>curl -u ***@my.domain:password --url imaps://email.my.domain/ -X
"EXAMINE INBOX"
* 88 EXISTS
* 0 RECENT
* FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
* OK [PERMANENTFLAGS ()] Permanent flags
* OK [UIDVALIDITY 1373] UIDVALIDITY value
* OK [UIDNEXT 332593] The next unique identifier value
http://stackoverflow.com/questions/12490648/imap-fetch-subject
I generated this command line:
C:\>curl -u ***@my.domain:password --url imaps://email.my.domain/Inbox
-X "FETCH 1:3 BODY.PEEK[HEADER.FIELDS (SUBJECT)]"
* 1 FETCH (BODY[HEADER.FIELDS (SUBJECT)] {23}
* 2 FETCH (BODY[HEADER.FIELDS (SUBJECT)] {44}
* 3 FETCH (BODY[HEADER.FIELDS (SUBJECT)] {23}

Notice how I'm seeing my command echoed back with the byte count of
the answer, but not the answer itself (i.e. not the subjects being
requested). Here's the results with the verbose "-v" switch, which
shows that the server is responding with the requested subjects!

A004 FETCH 1:3 BODY.PEEK[HEADER.FIELDS (SUBJECT)]

< * 1 FETCH (BODY[HEADER.FIELDS (SUBJECT)] {23}
* 1 FETCH (BODY[HEADER.FIELDS (SUBJECT)] {23}
< Subject: Re: Script
<
< )
< * 2 FETCH (BODY[HEADER.FIELDS (SUBJECT)] {44}
* 2 FETCH (BODY[HEADER.FIELDS (SUBJECT)] {44}
< Subject: RE: Exchange - Storage juggling
<
< )
< * 3 FETCH (BODY[HEADER.FIELDS (SUBJECT)] {23}
* 3 FETCH (BODY[HEADER.FIELDS (SUBJECT)] {23}
< Subject: Re: Script
<
< )
< A004 OK FETCH completed.
* Connection #0 to host email.my.domain left intact

This website: http://curl.haxx.se/mail/lib-2010-06/0019.html
reports that "Download all email in INBOX for this user with IMAP:
imap://user:password_at_mail.candelatech.com/INBOX?ALL"

But that doesn't work for me, either.

C:\>curl -u ***@my.domain:password --url imaps://email.my.domain/INBOX?ALL
* LIST (\Marked \HasNoChildren) "/" INBOX


My question is, other than using verbose output, how do I get curl to
return the actual fields requested not just the echo of my request
with the byte count of the result?

[Steve Holme]

Post by Stephen Jarjoura
I only recently learned that cURL supports both SMTP(S) and IMAP(S)
and have some ideas for using it in my scripting.

Curl has supported IMAP, POP3 and SMTP since 7.20.0 (February 2010),
however, its IMAP implementation has been quite lacking until about 12
months ago and even now there are still things missing!

Post by Stephen Jarjoura
While I've gotten the SMTPS to work well, I'm having some problems
with the IMAPS, specifically, it echoes back the command and a byte count
instead of returning the requested data.

Are you simply trying to obtain a list of subject lines for all the messages
in the Inbox or are you wanting to do other things as well as you gave a few
different examples - so I started to get a little lost with what you are
trying to achieve :(

For example:

* FETCH a specific message (or range of messages)
* LIST the contents of a mailbox
* SEARCH for new messages, or
* SEARCH for messages that match a certain criteria

Post by Stephen Jarjoura
--url imaps://email.my.domain/Inbox -X "FETCH 1:3 BODY.PEEK[HEADER.
FIELDS (SUBJECT)]"

Custom commands in IMAP "currently" override a list command so only untagged
responses are returned to the user. There is presently some discussion,
about this, going on over on the libcurl mailing list:

http://curl.haxx.se/mail/lib-2014-04/0067.html

As such this will be possible in the future but isn't with v7.36.0 :(

Post by Stephen Jarjoura
This website: http://curl.haxx.se/mail/lib-2010-06/0019.html
imap://user:password_at_mail.candelatech.com/INBOX?ALL"
But that doesn't work for me, either.

Unfortunately, and this is before my involvement with curl, but I believe
that was Ben's own fork and I must admit I haven't looked at his work there
:(

The query string isn't something that is currently supported in v7.36.0,
however, I have added basic support for it in v7.37.0 - as of commit
84c0aabe65 in the repo.

However, I would be interested to know what you are expecting such a URL to
do? For example: return a list of messages that match the query string or
fetch those messages as well as I may need to expand that support?

I appreciate with that URL Ben's example downloaded all messages but that
syntax isn't (as far as I know) mentioned in RFC5092 which is what we are
now trying to follow for our IMAP URL syntax:

http://curl.haxx.se/rfc/rfc5092.txt

Kind Regards

Steve
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html

[Stephen Jarjoura]

Post by Steve Holme
Are you simply trying to obtain a list of subject lines for all the messages
in the Inbox or are you wanting to do other things as well as you gave a few
different examples - so I started to get a little lost with what you are
trying to achieve :(

My initial goal was to try and get a list of subjects of messages in
an Inbox. But while I tried different iterations of the URL and
parameters, I kept getting the same sort of results. cURL returns the
first line the server responds with, which is an echo of the command
submitted, with a byte count on the end. It does not return any of the
subsequent lines that the server sends, which are visible with the
verbose flag. While my initial goal was to see a list of subjects, I
noticed that despite the documentation, I never did get a listing of
email messages, etc.

Post by Steve Holme

Post by Stephen Jarjoura
This website: http://curl.haxx.se/mail/lib-2010-06/0019.html
imap://user:password_at_mail.candelatech.com/INBOX?ALL"
But that doesn't work for me, either.

Unfortunately, and this is before my involvement with curl, but I believe
that was Ben's own fork and I must admit I haven't looked at his work there
:(
The query string isn't something that is currently supported in v7.36.0,
however, I have added basic support for it in v7.37.0 - as of commit
84c0aabe65 in the repo.
However, I would be interested to know what you are expecting such a URL to
do? For example: return a list of messages that match the query string or
fetch those messages as well as I may need to expand that support?
I appreciate with that URL Ben's example downloaded all messages but that
syntax isn't (as far as I know) mentioned in RFC5092 which is what we are
http://curl.haxx.se/rfc/rfc5092.txt

I did not know there was an RFC on a URL syntax for IMAP, that would
be very useful!
Since I am just learning how to use cURL for this purpose (i.e.
send/receive email) I am more than happy to learn whatever the correct
usage syntax is. Part of my frustration is that the documentation I
found reported results I could not reproduce. If one command format
reports "this will list messages in a mailbox" and then I only get a
list of flags for each message or I get a list of subfolders within a
mailbox, but not the messages themselves, then I'm left scratching my
head.

Post by Steve Holme
Kind Regards
Steve

Thank you, Steve!
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html

[Steve Holme]

My initial goal was to try and get a list of subjects of messages in an
Inbox.

If the only way to do that is via a FETCH command then that isn't going to
be possible with the current version of curl unfortunately :(

As I mentioned in my last email we are attempting to resolve that at
present, and I appreciate that doesn't help you archive what you want now.

But while I tried different iterations of the URL and parameters, I
kept getting the same sort of results. cURL returns the first line the
server responds with, which is an echo of the command submitted,
with a byte count on the end. It does not return any of the subsequent
lines that the server sends, which are visible with the verbose flag.

Yes - that is a known limitation of the current custom request
implementation in IMAP and unfortunately my fault !!

While my initial goal was to see a list of subjects, I noticed that

despite

the documentation, I never did get a listing of email messages, etc.

I would be interested to know what documentation you are referring to. I
wrote a fair amount of the email related documentation on the curl website
especially the URL syntax sections and libcurl code examples so if there are
errors in any of that then please let me know and I will correct it.

However, I wouldn't recommend you trust old threads in the archives (Prior
to January and Feburary 2013) relating to IMAP - especially those that refer
to someone's forked code, or articles on other websites that don't give curl
specific examples.

Post by Steve Holme
http://curl.haxx.se/rfc/rfc5092.txt

I did not know there was an RFC on a URL syntax for IMAP, that would
be very useful!

If you're not already aware of it, you may also find RFC3501 useful as that
describes in detail the IMAP protocol:

http://curl.haxx.se/rfc/rfc3501.txt

Since I am just learning how to use cURL for this purpose (i.e.
send/receive email) I am more than happy to learn whatever the
correct usage syntax is. Part of my frustration is that the
documentation I found reported results I could not reproduce.

As I mentioned above, if there is documentation that is incorrect or not
detailed enough then please feel free to highlight it and I will correct any
false information and try my best to add more detail where necessary.

If one command format reports "this will list messages in a mailbox"
and then I only get a list of flags for each message or I get a list of
subfolders within a mailbox, but not the messages themselves,
then I'm left scratching my head.

Unfortunately IMAP isn't as straight forward as POP3. For example: A LIST
command in POP3 will list the message IDs and size of each message for that
mailbox, as there is no concept of a folder hierarchy, but in the IMAP world
this command simply lists the folders in the root folder (mailbox) or
sub-folders of a folder.

To list messages one would typically SEARCH for them in IMAP. For example
"SEARCH NEW" or "SEARCH SUBJECT something". Once you have the result set
(list of UIDs) you can then perform the relevant FETCH commands to retrieve
those messages, however, due to the current limitation of not being able to
perform custom FETCH commands you will have to use the URL syntax to
retrieve them :(

Thank you, Steve!

You're welcome - please feel free to post any further questions here and I
will endeavour to answer them.

Kind Regards

Steve
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html