Cumulocity IoT Feedback Portal
We are deploying devices in field that can cache the measurements and transfer them to the Cumulocity only at a predefined intervals. It means, that the measurements reach cumulocity with delay, which sometimes is more than 24 hours.
We need to export all data from Cumulocity to an external big data lake. Due to high volume of data, real time notifications is not ideal mechanism for this. We woudl like to export data every 2 hours and capture all measurements and events generated in these last 2 hours. This is not a problem with events, because we can export based on created timestamp (last 2 hours) and we capture also the events which were cached for longer than 2 hours and represent events in the further past.
For measurements this is not possible and if we filter measurements from teh last 2 hours, we will miss all measurements that devices cached for longer than 2 hours. These are not in the interval anymore, although they have been transfered from device to cumulocity just recently.
What woudl help is create timestamp in measurement and possibility to filter based on created timestamp on measurements API
Hi Peter, This has now been delivered as part of release 10.13. Regards, Jane.
Hi Peter, I thought you would like to know that the first part of supporting this will be delivered in the next release of Cumulocity IoT, this will include the creation timestamp. To extend the API, document and run performance tests we have a feature on our backlog,
Regards, Jane.
Hi Nikolaus,
I���m aware of DataHub, however in our case, the customer needs to have the data exported into their own system (not data lake). And they need to make sure, they have all data, even if it is delivered with delay (because device cache it). With the current API implementation they can not be sure, that they are not missing past measurements if they come delayed and are stored with timestamp in a range that has already been exported. They are fine with exporting events, because they export them by ���create��� timestamp and therefore they always have the latest created events, even if they are stored with timestamp in the far past.
Peter
Von: Software AG <3a3fc9518d54084cbef01881-trendminer@iad-prod1.mailer.aha.io>
Gesendet: Mittwoch, 26. Februar 2020 12:56
An: Aha!
Betreff: Nikolaus Neuerburg responded to idea C8YCORE-I-145 created timestamp in measurements
Hi Peter. Thanks for the feedback. I perfectly understand the use case and will discuss options to extend our APIs with the development team. Are you aware that the Cumulocity IoT DataHub product addresses exactly this use case for you? It is designed to select data to export and reliably schedule data exports. Data is stored in a format optimized for an efficient exploration (through e.g. BI tools) and training of analytical models on top of the data. If you are interested I am happy to facilitate a session where we present the Cumulocity IoT DataHub product to you in more detail.