Hello,
I did a simple script injecting 5000 lines equally spaced over 6 seconds every 5 seconds… Meaning that there is an overlap of 1 second provoking out-of-order timestamp.
I don’t find any parameters that would avoid QDB to write directly data to disk provoking lots of copy/merge of the partition.
I though that by setting cairo.o3.min.lag=10000, it would buffer over 10 seconds and therefore reorder before writing to disk, but apparently not.
Any hint to avoid excessive disk usage in such case ?
Spreading over less than 5 seconds, and therefore avoid any O3 injection gives highly performant results.
Hi @Gabriel ,
That config option may only apply for ILP/TCP.
In general, the database will automatically impose a small commit lag when receiving O3 data to help reduce IO impact.
If you know ahead of time you will receive out of order data, you could aggregate and sort it before sending it to QuestDB.
Otherwise, maybe we can improve configurability for this use case.
Hi @nwoolmer ,
Thank you for the answer. I didn’t saw it was applicable to ILP/TCP only and since ILP/HTTP is now recommended, we are using HTTP. The behavior is less impacting IO’s with TCP.
Our processes are naturally generating these almost bounded out of order data… So we concluded that we need to buffer them. But since there was a parameter in QDB, I hoped it would do it without having to reimplement it.
If at some point you make it available for HTTP (and ideally Posgre also), that would be a nice feature because otherwise, it is really lots of disk writes.
Hi @Gabriel ,
I will raise it. Its worth pointing out that a min lag of 10 seconds wouldn’t be equivalent to a commit every 10 seconds. New inserts would still be buffered for 10 seconds individually, I think.
This is solvable client-side in the meantime by batching, and that will allow you send data (and therefore commit it) periodically i.e with 10 second intervals between commits.
In general though, there are some options that could be improved around ILP/HTTP, for example, having a dedicated threadpool as ILP/TCP used to have.