emap-go: a high-fanout IMAP client for Go

Introduction

A huge amount of automation comes down to one boring step: wait for an email, read a code out of it. Account verification, login OTPs, "confirm your address" links, order receipts. One inbox — usually a catch-all like *@yourdomain.com — feeds hundreds or thousands of concurrent tasks, each waiting for the one message addressed to it.

The naive way to do this in Go is to reach for an IMAP library and open a connection per task. It works on your laptop with three tasks. It falls apart in production, and it falls apart in three specific, predictable ways:

• Connection caps. Every IMAP provider limits simultaneous connections per account. Gmail allows 15. Open a connection per task and task #16 gets [ALERT] Too many simultaneous connections — your fleet wedges itself.
• Backlog on connect. A catch-all that's been live for a year has hundreds of thousands of messages in it. A lot of clients want to sync or enumerate the mailbox before they're useful — seconds to minutes of latency and tens to hundreds of MB of traffic, every time you connect, to read mail you don't care about.
• Fragile long-lived connections. IMAP IDLE connections drop. A client that doesn't reconnect cleanly — or reconnects in a tight loop with no backoff — either misses mail or gets the account rate-limited.

So we wrote emap-go: a small, dependency-free IMAP client built specifically for the high-fanout case. This post explains what it does differently, walks the architecture, and then puts it head to head against emersion/go-imap — the de-facto Go IMAP library — with reproducible benchmarks. Every number below comes from code you can run yourself: status403com/emap-go-bench.

emap-go is deliberately narrow. It receives mail from inboxes you own, fast and at scale. It does not send mail, manage folders, or implement the full IMAP surface. If you need a complete IMAP toolkit, emersion is excellent and you should use it — more on that below.

The one-line version

The core promise is simple:

One persistent IMAP connection per credential, fanned out to N subscribers — regardless of how many tasks are running.

Two thousand tasks reading a shared catch-all is two thousand subscriptions over one TCP connection, one IDLE loop, and a constant-size memory footprint. Not two thousand connections.

The API

You create a Manager, then Subscribe with a credential and a filter. Subscribing to the same credential again reuses the existing connection:

mgr := emap.NewManager(emap.DefaultLinger)
defer mgr.Shutdown()

cred := emap.Credential{
    Host:     "imap.gmail.com",
    Port:     993,
    UseTLS:   true,
    Email:    "catchall@example.com",
    Password: "app-password", // Gmail: use an app password
}

// 2000 tasks each watch their own per-task address on the shared inbox.
// Result: ONE IMAP connection, ONE IDLE loop, 2000 subscriptions.
for i := 0; i < 2000; i++ {
    addr := fmt.Sprintf("task%d@example.com", i)
    sub, err := mgr.Subscribe(cred, emap.Filter{To: addr})
    if err != nil {
        log.Fatal(err)
    }
    go func() {
        defer sub.Close()
        for msg := range sub.Ch {
            handleVerification(msg) // msg.To, msg.From, msg.Subject, msg.Body
        }
    }()
}

A Filter is how a subscription narrows what it receives. To is the important one — it's indexed for O(1) dispatch — but you can also gate on sender and age:

type Filter struct {
    To           string        // recipient, case-insensitive; "" = wildcard
    FromContains string        // substring match on From:
    Since        time.Time     // drop mail older than this
    MaxAge       time.Duration // drop mail older than now-MaxAge at dispatch
}

Most real code is task-scoped and context-driven, so there's a thin Mailbox wrapper that ties a subscription's lifetime to a context.Context — no manual Close():

mb := emap.NewMailbox(mgr, cred)

ch, err := mb.Watch(ctx, emap.Filter{
    To:           taskAddr,
    FromContains: "no-reply@vendor.com",
    MaxAge:       5 * time.Minute, // a verification mail is useless after 5 min
})
if err != nil {
    return err
}

select {
case msg := <-ch:
    code := extractCode(msg.Body)
    // ...
case <-ctx.Done():
    return ctx.Err() // subscription auto-closes
}

And for the "does this password actually work?" UI flow, there's a one-shot login that doesn't touch the pool:

if err := emap.TestLogin("imap.gmail.com", 993, true, email, password); err != nil {
    // bad credentials / unreachable host
}

That's the whole surface. The interesting part is what happens underneath.

Architecture

emap-go is four layers, each with one job.

Manager — the pool. It keeps a map of sessions keyed by (host, port, email) (email lowercased, password deliberately excluded so rotating a password reuses the same entry). Subscribe either creates a session or attaches to an existing one and bumps a refcount. When the last subscriber on a session leaves, the session doesn't drop immediately — it lingers (60s by default), so a task that unsubscribes and another that subscribes a second later share the warm connection instead of paying for a fresh LOGIN.

session — one per credential. This is the state machine: it owns the connection, the subscriber registry, and the watch loop. Subscribers are stored two ways — a flat set, and an index keyed by Filter.To (the empty key holds wildcard subscribers). Dispatch is therefore two map lookups regardless of whether the inbox has 5 subscribers or 5,000.

imapConn — one socket. The raw wire protocol: a single net.Conn, commands serialized through a mutex (one in flight at a time, as IMAP requires), literal-aware response parsing, and an IDLE implementation that writes DONE concurrently with the read loop.

The watch loop. On connect the session checks CAPABILITY. If the server advertises IDLE, it runs an IDLE loop (push delivery, ≤1s latency, re-issued every 25 minutes to stay under the RFC's ~29-minute ceiling). If not, it falls back to polling every 30 seconds. Either way the cadence is per credential — two thousand tasks on five inboxes is five IDLE loops, not two thousand.

Three design decisions do most of the work, so they're worth pulling out.

1. Pool by credential, fan out to subscribers

This is the headline. A per-task client model scales connections linearly with task count and detonates at the provider's connection cap. emap-go holds the connection count flat at the number of distinct inboxes, and treats tasks as cheap in-process subscriptions.

Dispatch is non-blocking with drop-oldest semantics: each subscriber has a small (8-message) buffered channel, and if a consumer stalls, the oldest queued message is dropped rather than blocking the whole pipeline. A hung task can't pin memory or stall delivery for everyone else.

2. Skip the backlog with UIDNEXT

When emap-go issues SELECT INBOX, the server replies with, among other things, * OK [UIDNEXT N] — the UID the next message will get. emap-go records N, sets its watermark to N-1, and from then on only ever issues UID FETCH N:* — strictly above the watermark. On a fresh connect that matches zero messages.

The result: an inbox with a million messages in it costs exactly the same to attach to as an empty one. emap-go never downloads backlog it didn't ask for. The watermark survives reconnects, so mail that arrives during a disconnect window is still picked up on resume.

There's a deliberate safety guard here too: if a non-conformant server omits UIDNEXT, emap-go refuses to FETCH at all rather than fall back to FETCH 1:* and accidentally pull the entire mailbox into memory. Delivering nothing beats OOM-ing on a million-message inbox.

3. Reconnect like you mean it

Long-lived IMAP connections fail. emap-go distinguishes two cases:

• Transport failure (broken pipe, EOF, server drop): reconnect with exponential backoff — 1s, 2s, 4s, … capped at 60s — until it succeeds or the context is cancelled. After reconnecting it forces a FETCH so any mail from the outage window is delivered before resuming IDLE.
• Protocol rejection (the server returns BAD/NO to IDLE): it exits the loop cleanly instead of reconnecting. A fresh connection would hit the same rejection, so hammering the server is pointless and just burns connection slots.

That distinction — "is this connection sick, or is the server telling me no?" — is exactly the kind of thing you end up hand-rolling on top of a lower-level client, and getting subtly wrong.

How it compares to emersion/go-imap

emersion/go-imap is the standard Go IMAP library — ~2.3k stars, and what most projects reach for. It comes in two flavours: v1 (IMAP4rev1, stable, last released 2022) and v2 (IMAP4rev2, currently in beta). It is a genuinely good, RFC-complete toolkit that implements both a client and a server.

And that's the key framing for everything below: emersion and emap-go are not the same kind of thing.

emersion gives you correct IMAP primitives and lets you build anything. emap-go makes one opinionated decision — pool and fan out — and hands you the result. So the honest comparison isn't "which library is better"; it's "what does it cost to build an N-task mail-watcher with each?" With emap-go you get pooling, fanout, backlog-skip, and reconnect for free. With a low-level client the idiomatic answer is one client per task — and that's what the benchmarks measure.

Where a careful emersion user could match emap-go (for example, reading UIDNext from the SELECT response manually instead of syncing the mailbox), we say so explicitly. But the connection pooling, fanout dispatch, IDLE lifecycle, reconnect state machine, and backpressure are genuinely hard to replicate — that's roughly 1,000 lines of carefully synchronized Go you'd have to build and maintain on top of any per-connection client. The point of emap-go is that it does all of this by default; the point of the benchmarks is to show what the default, idiomatic approach actually costs.

Benchmarks

Methodology

Real numbers, reproducible, no hand-waving:

• Both libraries are driven against the same tiny localhost IMAP server (~250 lines, included in the bench repo). No external services, no real mailboxes, no network flakiness.
• One OS process per measurement. The server runs in its own process; each client run is its own process. Memory (VmRSS from /proc/self/status) and goroutine counts therefore reflect the client library alone, after a forced runtime.GC().
• Connections are counted server-side — every accept() is tallied — so the numbers don't depend on trusting either client's self-report.
• Startup volume is counted server-side too: the exact number of FETCH responses emitted and bytes written on the wire.
• Go 1.26, plain TCP on 127.0.0.1, 16-core box. emersion is v2 (imapclient).

The whole suite runs in under a minute: ./sweep.sh && node charts/gen.mjs.

Connections: the wall everyone hits

N task-watchers, one shared inbox. We count how many IMAP connections each approach actually opens.

emap-go holds one connection no matter what. The per-task client model opens one connection per task — and crosses Gmail's 15-connection-per-account cap at N=16. In production, against a real provider, the emersion-per-task design simply cannot run this workload on a single account; it errors out at task 16. The benchmark lets it keep going only because the localhost test server imposes no cap.

Memory and goroutines

Same workload, measuring process RSS and goroutine count as N grows.

emap-go stays at 3 goroutines and ~10 MB whether it's serving 1 task or 2,000 — the only thing that grows is the tiny per-subscription bookkeeping. The per-task model scales connections, goroutines (~2 per client), and memory linearly. At 2,000 watchers that's 4,001 goroutines and ~5× the memory, for the same mail.

Startup cost vs inbox backlog

How long until the client is ready to receive new mail, and how much does it download to get there, as a function of how much mail is already in the inbox?

Important caveat: These numbers compare emap-go's default behavior against emersion's naive path — syncing the mailbox before fetching. A careful emersion user can read UIDNext from the SELECT response and skip the sync, getting the same constant startup. The difference isn't capability — emersion exposes the same primitive — it's what happens by default. emap-go does the UIDNEXT skip automatically and refuses to FETCH at all if the server omits it, rather than risking an accidental full-mailbox pull.

emap-go's startup is constant — ~1 ms and 0 backlog messages, whether the inbox holds a thousand messages or a million. The naive mailbox-sync path pays linearly: on a 1M-message catch-all that's 2.5 seconds and ~102 MB before the client can do anything useful. But again — this gap is about defaults, not limits. The startup benchmark is really measuring "what does the library do if you don't think about it." The connection and memory benchmarks above are the ones where the architectural difference can't be closed by a smarter caller.

Supply chain

Not a runtime metric, but it matters for auditability and for how much you have to trust:

emap-go pulls in zero external modules — it's standard library plus net/mail, about 1,600 lines of Go in a single package you can read in an afternoon. The emersion v2 client path pulls in 13 modules and 37 go.sum entries to vet. (Static binary sizes are comparable, ~4 MB each — this is about dependency surface, not binary size.) Neither is "right"; a full toolkit earns its dependencies. But for a narrow, long-running, credential-handling component, a small auditable surface is a feature.

When not to use emap-go

Being narrow cuts both ways. Reach for emersion (or another full client) instead when you need:

• To send mail, manage folders, copy/move/expunge, or anything beyond receiving from INBOX. emap-go watches INBOX and nothing else.
• Full mailbox sync or search — building a mail client UI, indexing an archive, server-side SEARCH. emap-go is built to skip the backlog, which is the opposite of what you want there.
• Rich message parsing — MIME trees, attachments, nested multiparts. emap-go hands you decoded headers (To/From/Subject/Date) and the raw TEXT part, which is exactly enough to regex a code or a link out, and no more.
• IMAP4rev2-specific features or a server implementation. That's emersion's home turf.

emap-go is the right tool when the sentence "many tasks need to read mail from a few inboxes, fast, forever" describes your problem. For most other IMAP work, it isn't.

Key takeaways

• The high-fanout mail problem — many tasks, few shared inboxes — breaks the connection-per-task model on connection caps, backlog downloads, and fragile reconnects.
• emap-go pools one connection per credential and fans it out to N in-process subscribers: 2,000 watchers = 1 connection, 3 goroutines, ~10 MB, versus 2,000 connections / 4,001 goroutines / ~49 MB for the per-task approach.
• It skips the backlog via UIDNEXT by default, so startup is constant (~1 ms, 0 bytes of backlog) regardless of inbox size. A careful emersion user can do the same manually — the difference is emap-go does it automatically and fails safe when the server misbehaves.
• It ships with IDLE-first delivery, polling fallback, exponential-backoff reconnect, and drop-oldest backpressure built in — the resilience you'd otherwise hand-roll on a lower-level client.
• It has zero external dependencies — ~1,600 lines of stdlib Go — at the cost of being deliberately narrow: receive-only, INBOX-only, no sending.
• emersion/go-imap is the right call when you need a complete IMAP toolkit; emap-go is the right call when you need this one workload to scale.

Source: status403com/emap-go. Run the benchmarks yourself: status403com/emap-go-bench.

Scaling automation like this is part of the reverse engineering and automation work we do as a service. Need high-throughput automation built properly? Get in touch.