IMAP stands for Internet Message Access Protocol, and it’s an open standard for accessing messages in an email account.
While IMAP is a crucial aspect of receiving emails, it’s not always easy to set up (or comprehend).
In a nutshell, IMAP is the protocol used by email clients such as Mail.app, Thunderbird, and Mailspring to download messages from your email account and perform actions such as archiving or sorting them into folders.
A Brief History of IMAP
Mark Crispin, a Stanford professor, invented IMAP in the 1980s.
The IMAP specification is in the form of a “Request for Comments,” or RFC. It is essentially a letter outlining how to implement the protocol that the Internet Engineering Task Force has adopted as a standard. RFCs are subject to change or revision, and the current RFC that describes IMAP is RFC3501. That’s right: the email-sync protocol used by most email clients is over 15 years old!
IMAP is a better alternative to the older POP protocol, with the goal of “permitting manipulation of remote mailboxes as if they were local”. POP’s mode of operation was to download mail to your local machine and delete it from the server, making it difficult to manage your mail if you used multiple computers.
This was not a problem when mainframes were used for timesharing. However it became one as personal computers grew more common, and you might have a PC at home as well as at work.
The early version of IMAP only enabled “online” operation․
Because messages were only identified by a “sequence” number. Sequence numbers were not permanent across client sessions, you couldn’t use it to sync a copy of your mailbox locally and then update your local copy when you rejoined. One can only imagine that in those days, people assumed that whenever they wanted to check their email, they would always have a solid network connection.
With the introduction of IMAP4 in 1994, support for “disconnected” actions added to IMAP.
This version of the protocol includes persistent message identifiers (known as “UIDs”). A complete draft spec describing how to use IMAP commands to sync changes made to locally cached data to data stored on a server.
How IMAP Works
A web browser and a server, or an app and an API, are in many modern client-server protocols.
These APIs are all on top of HTTP (“hypertext transfer protocol”). It defines the semantics of various request types such as GET, PUT, and POST.
HTTP is on top of TCP (“transmission control protocol”), a lower-level protocol. It simply ensures that data packets reliably deliver to their intended destination.
IMAP, like HTTP, is on top of TCP and defines the semantics of various requests, referred to as IMAP commands.
IMAP is far more concise than HTTP in general. It was critical for IMAP in the early 1990s when network connections were extremely low-bandwidth.
Step by Step
The initial step for any client is to establish a connection with the remote server through a certain port. Because mail servers first supported plaintext connections and then added support for secure encrypted connections, these ports can change. You can usually find your provider’s settings by looking through their support articles.
On the command line, we can connect using the openssl command-line utility. We have to use openssl rather than telnet, because otherwise we could end up sending sensitive data like passwords and private emails in plaintext over the network, leaving them open to network sniffing by an attacker. openssl protects our connection by making a secure connection and verifying the remote server’s certificate before proceeding.
In the above command, s_client means we’re invoking openssl’s functionality to act as a basic SSL/TLS client, -connect specifies which hostname and port to connect to, and -crlf converts when you press “Enter” in your terminal to the characters expected by the IMAP server to properly end a line. After connecting, openssl checks the SSL/TLS certificate on the server to make sure our connection isn’t being hijacked, and the IMAP server displays a greeting that says it is ready to receive requests from the client.
Commands
IMAP commands generally look like this:
We’ll explain these parts in the context of an example command, called CAPABILITY. CAPABILITY allows you to ask a server which IMAP extensions it supports:
A command is equivalent to an HTTP request. It’s instructing the server what to do or requesting information from it.
We’re asking the server to tell us the capabilities it supports in this scenario.
In a moment, I’ll go over capabilities in further detail.
The CAPABILITY command doesn’t take any arguments, but if it did, we’d write them after the command name and before pressing enter.
Tagging
The client generates tags (tag1 in this example), which the server returns on the last line of the response to a command.
Tags can be made up of any alphanumeric letters and even symbols. They don’t have to be ascending integers—though it is a useful approach to make them distinctive.
Untaggged answers are server responses that begin with *. They indicate that they do not represent the fulfillment of a command requested by the client.
The capability list is the untagged response in this case. While the tagged response is the command’s OK status with a humorous remark from the server programmers.
This tagging capability allows a server to process several requests from the same client on the same connection at the same time and send back the appropriate tag to signify completion.
Many clients, in practice, do not support the ability to issue concurrent queries, and instead merely wait for data to come on the open socket when a command is delivered.
Capabilities
So, what exactly does this list of “capabilities” imply?
The term “capability” refers to a functionality that the server provides.
Some of these features, such as AUTH=PLAIN, are included into the core IMAP4rev1 specification, while others are enabled by extensions.
New commands can be introduced to the IMAP protocol using extensions rather than a new version of the protocol specification.
Because it’s pointless to display capabilities that aren’t relevant to the logged-out or logged-in states, the list of supported capabilities on a server will be different before and after signing in.
Authentication
Here’s how you authenticate your mail account to an IMAP server:
Server Responses
As you can see, server responses don’t always say OK. Here’s a quick guide to the allowed server responses to a command:
OK- success!
NO- failure!
BAD-I didn’t understand what you said!
The server is free to add a bit more information after the response if it so desires, like [AUTHENTICATIONFAILED] Invalid Credentials (Failure).
As you can see, the server provides the client the list of capabilities again after a successful login, indicating which new capabilities unlocked as a result of the login and which old ones removed.
UIDPLUS, MOVE, and CONDSTORE are some of the more common extensions on this list.
ESEARCH and X-GM-EXT-1 are two custom extensions that are supported by Gmail but not by other email clients.
Listing Folders
Let’s try to do something productive now that we’ve logged in. Let’s start by seeing which folders are there in the mailbox:
Whoops! As you can see, if you get the arguments wrong to a command, the server will reject it and say BAD client!
Folder Flags
Special flags for folders indicate in parenthesis. Some of these flags, such as HasNoChildren and HasChilden, are useful for traversing the folder hierarchy, while others are used to indicate particular properties of a folder, such as whether it includes drafts (Drafts) or sent messages (Sent).
These flags allow clients to adjust their behavior to the mailbox, such as saving sent messages to the appropriate folder regardless of the user’s mailbox language.
Sessions
Let’s have a look at some mails now.
Because IMAP connections are stateful, we must first use the SELECT command to open a session on the folder we’re interested in:
Here’s where the server sends us a lot of data!
IMAP relies heavily on sessions, which last from the time you SELECT a mailbox until you UNSELECT it, SELECT another mailbox, or log out.
When opening a session, this server has sent us, in addition to the OK indicating success:
- a list of active flags in use on messages in the mailbox
- a list of supported flags that could be used on messages in the mailbox
- the
UIDVALIDITY– more on this in a bit - the number of messages in the mailbox
- how many “recent” messages are in the mailbox, which means how many messages are tagged with the
Recentflag - the “predicted next UID”
- a
HIGHESTMODSEQvalue – more on this in a bit
A few of these items are going to require a bit more explanation. First, UIDs.
UIDs & Sequence Numbers
A reminder: IMAP2 existed once upon a time, and messages in IMAP2 recognized by sequence numbers.
Sequence numbers were exactly what they sounded like: a series of monotonically increasing numbers, one for each message in a mailbox:
If there were new messages when you selected a mailbox, there would be extra sequence numbers.
And, if you rearranged messages and started a new session, the sequence numbers allocated to messages may differ from those in the previous session.
If all you did was connect, read your mail online, and disconnect, this extremely simple technique worked quite well.
It didn’t work, however, if you wanted to connect, download all of your mail locally for offline access, and then reconnect later to have your client sync its local state with the server’s state—because non-persistent sequence numbers prevented clients from mapping local mailbox state to remote mailbox state, because messages on both sides could have different IDs!
Enter IMAP4 and UIDs. As long as the message remains in the same folder and the server’s UIDVALIDITY integer has not increased, UIDs (unique IDs) are durable.
When a client starts a new session on a mailbox, it must compare the server’s UIDVALIDITY to its local cached value, and if the server’s value is higher, it must discard all cached local UIDs and resync from the beginning.
Otherwise, it’s free to compare its locally cached data to the server’s data and only sync changes.
IMAP4 supports both sequence numbers and UIDs for backwards compatibility. And you may get UIDs by prefixing any command that returns message identifiers with UID:
Statefulness
While we’re on the subject of statefulness, here’s something to think about: Statefulness is not a desirable property in protocols in general.
Because the SELECT command can be slow, and every time a pool consumer checks out a new connection, it must guarantee that the correct mailbox is selected, statefulness makes using connection pools with IMAP more difficult.
Statefulness also means that application code must maintain track of the state and ensure that it is in sync with the connection’s real state, or else a new client-server back-and-forth will be required every time the information is needed.
Stateless protocols are generally easier to program than stateful protocols.
Fetching Message Data
OK let’s finally actually download data from a message!
IMAP has a number of knobs that allow clients to request partial data about messages, such as the size of the message, flags (such as Seen, which indicates that the message has been read, and Answered, individual headers, and individual MIME parts from within a MIME message, allowing clients to optimize their bandwidth usage by downloading only the data they require.
For example, a client can immediately download simply the basic headers for all messages in a folder to create a message listing for a user to browse through, and then fetch the message bodies on demand.
Unsolicited responses & IDLE
You may have noticed some strange untagged responses in the previous example:
* 55 EXPUNGE * 54 EXISTS This is an indication that one message has vanished from the mailbox and another has appeared while the mailbox has been open.
The server can deliver these unsolicited messages to a connected client at any moment under IMAP.
You can also place an IMAP connection into IDLE mode.
Disconnected syncing
Disconnected syncing enables email programs to function properly even while using airplane wifi or a poor cellular connection.
Because IMAP wasn’t designed to support it, it’s not so much a first-class feature of the protocol as something that can be built on top of it with some work.
In its original incarnation, the method for doing a detached sync with IMAP4 was rather straightforward: you’d compare the list of UIDs you have locally for each folder to the list of UIDs currently in the folder on the server, then retrieve new messages and delete old ones based on the differences.
As the number of mailboxes increased, this became increasingly difficult for clients to manage, and later extensions such as CONDSTORE and QRESYNC improved the flow for detached resyncs. However, even with these additions, implementing a detached resync is difficult—and many IMAP server implementations still don’t support the newer extensions, thus general-purpose clients must include conditional logic for different servers.
RFC4549 is a standard specification for implementing detached syncing on top of IMAP.
Click here to read more useful and interesting articles.
