Building and Scaling a Marketplace Monitoring Pipeline in Go

This case study explores the design and scaling of a monitoring pipeline for a large, well-established Central European marketplace, focusing on concurrency, memory efficiency and reliable data processing.

Collecting listings was only the beginning

A list of marketplace offers tells you what is available now. It does not tell you what changed, which sellers are gaining activity, or whether a product is attracting sustained demand.

We built our marketplace pipeline to keep listings useful over time. It runs keyword and seller monitors, revisits their results, identifies relevant changes and makes the collected information available for analysis.

The engineering challenge was keeping that work reliable as the number of monitors and the amount of data grew. Broad searches, frequent refreshes and detailed offer records put very different demands on the same machine.

These figures describe our operating experience and documented infrastructure. They are not a throughput benchmark or a guarantee for every combination of searches and refresh intervals.

The central decision: control how much work happens at once, preserve completed progress, and avoid repeatedly processing records that have not meaningfully changed.

How the system works

Continuous monitoring means revisiting familiar listings. Treating every cycle as a fresh collection job would repeat expensive processing and make restarts unnecessarily costly.

Our pipeline separates monitoring intent, active work and stored observations. A monitor defines the keyword or seller being followed. The execution layer handles recurring work. Persistent storage keeps the information needed to continue after interruptions and compare later observations.

Each monitoring cycle follows the same path:

1. Assign the work. The scheduler assigns a due keyword or seller monitor to a scraper process with available capacity.
2. Collect and compare. The monitor collects listing results and checks them against the saved state to identify new or changed offers.
3. Enrich selectively. Offers that need more detail enter bounded enrichment work, so unchanged listings do not trigger the same expensive processing on every cycle.
4. Save progress. Results are persisted incrementally, keeping the latest offer state available and preserving completed work if the cycle is interrupted.
5. Feed the analysis. Stored observations support comparisons across listings and sellers, including purchase-count movement and product activity over time.

That separation matters when a process restarts or capacity moves between machines. Work can be reassigned without treating everything previously collected as new.

We also separate historical changes from the latest known offer state. Looking up the current record and examining how a listing changed are different tasks; neither should require rebuilding the other from scratch.

This gives the pipeline a useful foundation for both operational monitoring and later analysis, while keeping recovery part of normal operation.

Scale comes from controlling work

Adding more monitors increases several kinds of load at once: collection, offer enrichment, data conversion and database activity. Allowing all of them to expand without limits would make a larger workload less predictable.

Our use case does not require constant refreshing. We use conservative request-rate limits and deliberate delays between collection cycles to avoid putting unnecessary pressure on the marketplace. Refresh intervals follow the freshness needed for our analysis, rather than how quickly the system could collect the data.

We control concurrency at more than one level. There are limits on active monitors within a process and on expensive enrichment work inside each monitor. Scheduling accounts for available capacity, while queued work waits for a slot.

Twenty-five configured monitors does not mean twenty-five operations running simultaneously. Some monitors are waiting for their next cycle; others are processing results or enriching selected offers. Their combined workload determines the pressure on the machine.

The practical decisions are straightforward:

• Limit expensive work. More parallelism is useful only while processing and storage can keep up.
• Skip unchanged records. Repeated observations should not automatically trigger full enrichment.
• Save progress incrementally. A failure late in a cycle should not discard everything completed earlier.
• Watch the backlog. Bounded execution still needs monitoring when incoming work exceeds available capacity.

The system can distribute work across scraper processes and recover assignments after failures. More machines provide capacity, but they do not replace those controls.

Large records make memory a concurrency problem

The number of listings alone is a poor predictor of memory use. A compact listing record and a fully enriched offer have different costs, and several representations of a record can exist during processing.

Parsed objects, owned data and encoded output may overlap in memory. Increase the number of records being processed concurrently and those temporary costs multiply. Limiting worker count alone does not fully control that pressure.

We address it through three decisions:

Streaming does not eliminate every allocation. Individual records still need to be parsed, transformed and encoded. The benefit is that an export does not need memory proportional to the complete exported dataset.

Likewise, concurrency limits do not make every retained object disappear. Monitoring state and records retained during a cycle still contribute to memory use. The goal is predictable resource use under the intended workload, measured alongside latency and failures, rather than an unsupported claim of zero-copy processing.

Reliability includes the quality of the observations

A monitor that silently stops is as problematic as one that crashes. We track active work, errors, latency, queue depth and newly seen or changed listings, so operational problems can be distinguished from a quiet marketplace.

Failures have bounded retry and recovery paths. Repeated failures can stop work temporarily rather than consume capacity indefinitely, and interrupted monitors can be reassigned from their saved state.

For analysis, those interruptions matter too. A gap in observations is not evidence of zero activity. A seller may continue trading while a monitor is unavailable. Reports need to retain that distinction instead of presenting incomplete coverage as a complete history.

From listing changes to sales monitoring

We have added sold-count tracking based on changes in observed purchase counts. It is working, with reliability testing ongoing.

Comparing observations can reveal movement that a single snapshot cannot: which offers are accumulating purchases and how that activity changes over time. But the interpretation depends on what the source count means.

Only compatible cumulative counts support a straightforward period delta. Rolling windows, resets, relisting and changes in scope can make subtraction misleading. Stock changes are a separate signal because sellers can restock or edit inventory.

Those distinctions determine whether a dashboard provides useful sales indicators or merely precise-looking numbers. Reliability testing needs to cover missing observations and changing counter behaviour as well as successful collection.

Using the data to identify trends

The pipeline gives an analysis layer observations linked to offers, sellers and, where available, catalog products. That supports two useful perspectives:

Trend analysis can combine purchase-count movement, price changes, newly observed offers and persistence across monitoring cycles. It also needs to avoid counting the same offer multiple times simply because several keyword monitors found it.

Broader coverage must not be mistaken for growing demand. Adding more monitors naturally discovers more offers. A useful comparison accounts for observation coverage and uses a consistent time window.

That ranking methodology sits downstream of collection. The pipeline supplies the data and identities; the choice of trend score and alert threshold depends on the business question being asked.

What this project demonstrates

The useful outcome is a continuously maintained information source, rather than a collection that loses value as soon as the first run ends.

For us, the important engineering work was making that continuity practical: preserving progress, controlling concurrency, reducing repeated enrichment and delivering large datasets without buffering them in full.

Those decisions connect the collection system to its purpose: following sellers, observing product activity and giving an analysis layer a reliable basis for comparison.

Legal disclaimer

This case study presents a high-level engineering analysis of a marketplace monitoring system designed for the operational demands of a major Central European e-commerce platform. Platform-identifying details have been omitted, and diagrams simplify the architecture for educational purposes.

This material is provided for general informational and educational purposes only and does not constitute legal advice. It does not grant permission to access third-party systems, collect data, or bypass access controls. Any implementation must comply with applicable laws, contractual obligations, intellectual property rights and data-protection requirements.

Need marketplace data you can monitor over time? We build web scraping and automation systems, including the backend and infrastructure that keep them running. Discuss your project with us.