How it all started

As I wrote before, we already had an industrial collector running.

It was reading data from the PLC, putting snapshots into a RAM queue, and then sending them to ClickHouse.

The main pipeline looked like this:

PLC
 ↓
collector
 ↓
ClickHouse
 ↓
history
 ↓
ML

Signals were collected continuously.

The time series kept growing.

We could already train models on it, compare different periods, look for event precursors, and experiment with different ML approaches.

Then the guys from the neighboring department started rolling out an MES production management system:

some of our data had to be sent to MES.

My initial solution was simple.

The data was already in ClickHouse.

Build an API, and MES can read whatever metrics it needs.

MES doesn't need even 1-second frequency anyway, so the idea of just "pulling data from ClickHouse" seemed perfectly logical.

For about a month, everything looked fine.

And then the chaos started:

Add this register too.

Then:

We also need these sensors.

Then:

Oh, and we need one more metric here.

And every one of these "small" requests forced me to rebuild parts of the ML time-series structure.

At that point it became obvious that we were building an architectural landmine for whoever had to deal with this later.

Why MES and ML couldn't live under the same contract

For MES, changing the set of signals is completely normal.

MES needs a current picture of production.

Today the business wants to see temperature.

Tomorrow a counter.

The day after tomorrow some other process parameter.

The dataset naturally evolves.

For ML, the situation is almost the opposite.

I need a stable time series.

Let's say yesterday a snapshot looked like this:

[A, B, C, D]

Today:

[A, B, X, C, D]

And a week later:

[A, B, X, C, Y, D]

Technically, I still have the data.

But now I also have to maintain some kind of instructions saying:

Okay, before this date index 3 meant one thing, after that date it meant something else, and somewhere around here the decoding logic also changed...

And eventually, instead of analyzing data, you start dancing with a tambourine just to figure out who changed what and when.

For ML, changing the historical data contract is CRITICAL.

Especially when you're working with time series.

So either every time the catalog changes we throw away the "past", or we stop abusing our time series and separate the storage.

The main rule: nobody should be able to change the ML dataset

In my case, MES should not be able to change the ML dataset. At all.

The ML catalog should live its own life.

If I decide to change it, that becomes a separate, deliberate NEW version of the dataset.

The MES catalog, on the other hand, should be free to grow every week if necessary, because the MES customer — management — seems to have a new vision of the system every time.

So now the collector had to be split into two data flows.

                ┌→ ML
PLC → collector ┤
                └→ MES

And that's where the main problem appeared.

Let's just run two collectors and call it a day

The obvious solution looks like this:

PLC ← collector ML

PLC ← collector MES

Architecturally, it even looks pretty reasonable:

  • two independent applications;
  • two independent schedules;
  • two catalogs.

Nobody gets in anyone else's way.

Except our data source is not Kafka or some cloud database.

The source is a real industrial PLC.

And the idea of putting two independent processes on it, each hammering the controller on its own schedule and both working through the same TCP port... I really don't want to imagine that.

Not because it's impossible.

But because experiments like that can have very expensive consequences: unstable PLC operation, machine downtime, or even equipment damage.

Especially considering that the ML collector is already running fairly close to the limit of its one-second cycle.

So: one owner of the PLC, two different consumers

The final architecture now looks like this:

                    ┌→ ML catalog
                    │      ↓
PLC → one reader ───┤   ML queue
                    │      ↓
                    │  ML ClickHouse
                    │
                    └→ MES catalog
                           ↓
                       MES queue
                           ↓
                      MES ClickHouse
                           ↓
                         MES API

The main idea:

one reader owns the TCP connection to the PLC.

It alone decides:

  • when to read;
  • what to read;
  • in what order;
  • how much time can be spent on it.

After that, the data is split into two independent logical pipelines.

ML has its own YAML catalog.

MES has its own one — and now it can be changed however we want, safely.

Its own queue.

Its own database.

Its own storage schema.

Roughly speaking: if MES asks for another bunch of signals tomorrow, the ML catalog won't be affected at all.

Another limitation — the weak mini PC connected to the PLC

Yes, yes.

The edge computer connected to the PLC is, let's say, not exactly powerful.

And now it has to handle both the MES and ML catalogs.

Well, more precisely, that was my original idea.

I initially wanted the reader to build two complete snapshots:

ML snapshot
MES snapshot

But that meant extra data structures, array copying, and more work inside the most sensitive process of all — the reader, which has to finish polling the PLC on time.

And the cycle isn't exactly fast for industrial equipment — we're working with roughly one second.

After a couple of attempts to run the modified pipeline, the Front Compact — that same edge computer — consistently went into OOM, out of memory.

So I had to change the reader structure, rethink the cache, and simplify the catalogs almost down to raw values:

A, B, C, D, E...

Why not just put a local database on the edge PC?

Well, the obvious answer would be: install a local database directly on the Front Compact and forget about the whole problem.

And yes, that would be convenient:

PLC → local DB → everything else

But the Front Compact has a regular SSD inside, and sooner or later constant writes would turn it into a consumable.

So intermediate data had to stay in RAM queues.

Persistent storage lives in ClickHouse on another node.

In other words, the edge should remain as dumb as possible — sorry:

read
 ↓
put into RAM
 ↓
send
 ↓
forget

With this architecture, we really don't need to buy expensive hardware for every machine.

And the value of powerful hardware at this layer becomes minimal anyway.

The collector is simple and can run reliably even on low-powered PCs.

What we ended up with

After splitting the flows, the ML contract stayed intact.

It still contains 7,271 logical values, and most importantly, the structure of the ML catalog did not change because MES appeared.

The historical database also stayed exactly as it was.

Nothing had to be migrated or rewritten retroactively.

MES, meanwhile, got its own catalog with 641 signals and its own ClickHouse database.

So now the MES catalog can be changed independently from ML, and vice versa.

What was all this for? The main insight

The biggest takeaway here isn't really about PLCs or ML/MES.

It's about learning data engineering from my own mistakes.

And the longer I work, the more I realize how little I actually know :)

Queues, cache, RAM queues, constant OOM on weak hardware, catalog headaches, time-series data consistency, and so on.

So, as Uncle Vladimir Ilyich used to say:

"Study, study, and study again!"

Because nothing is ever perfect.

That last part is from me.