thingsboard

Open-source IoT Platform - Device management, data collection, processing and visualization.

dashboard java cloud coap iot iot-analytics iot-platform iot-solutions kafka lwm2m microservices middleware mqtt netty platform snmp thingsboard visualization websockets widgets

History

Viacheslav Klimov 86b5378d59 EDQS (#3196 ) * Experiments with CSV * CSV Loader v1 * EDQ tests * Volatile variables instead of final * Improvements * updated loader with new entities * Fix double memory usage issue * Basic data structures and load * Minor improvements * Snappy + Large String reuse * added EntityFields classes for each entity * Basic implementation * Minor improvements to KeyFilters * implemented RepositoryUtils.checkKeyFilters * Generic query implementation * New structure * Refactoring and few processors implementation * extended DeviceData with shared/client attributes and device profile * Minor refactoring of attribute scopes * DeviceTypeFilter support * Strong types of fields for each entity data class * DeviceType and AssetType filters * EntityView and Edge queries * Relations Query * Relation Query Implementation * Update EDQS module version * Sync with EDQS via Kafka * EDQS: major refactoring * EDQS API requests via Kafka * EDQS: full sync with the database * Refactoring for EDQS sync * EDQS: major refactoring and new features * EDQS refactoring, count query support, fix tests * EDQS: refactoring for query processors * Fix EDQS pom version * Cleanup edqs.yml * EDQS: tenant partitioning strategy; refactoring * EDQS: latest events queue * EDQS: support for monolith setup; RocksDB; other improvements * EDQS: merge sync and events topics, introduce state topic * EDQS: dynamic repartitioning * implemented entity data query filters for edqs * EdqsEntityQueryControllerTest - use in-memory queue * edqs-filter fixes, added test * EDQS: blob entity support * EdqsEntityQueryControllerTest - use in-memory queue * Use DummyEdqsService when disabled * Fixes for EDQS * Refactoring for EDQS tests * Fix edqs requests partitioning * EDQS: Fix for attributes handling * Fix attributes saving in EntityServiceTest * EDQS: refactoring, fixes * Minor refactoring for query processor * added ownerName/ownerType support * fixed relation query processor * fixed EntityServiceTest * refactoring * added support for parentId for relation query result * Get rid of EntityNameFetcher * Add fixme for relation query processor * db restore with select all edqs fields * fixed entity deletion * fixed FieldUtils with new EntityFields * dao method renamed * EDQS: instance groups with same partitions; automatic sync; multiple fixes * Refactoring for EDQS sync * EDQS: refactoring * Fix startup with Kafka * fixed EntityQueryControllerTest * fixed EdqsEntityServiceTest * Separate queue admin for EDQS request template * Implement new EDQS partitioning strategy * EDQS: multiple fixes and refactoring * Add mock EdqsRocksDb beans to tests * added edqs stats for inmemory/grafana * fixed filter tests * Update todos * Refactoring for QueueConfig * Improvements and refactoring for EDQS consumers * implemented TODOs * test fixes * Consume state topic up to end offsets * edqs stats refactoring * EDQS: cleanup on partitions removal; refactoring * EDQS: minor refactoring * EDQS: remove CSV loader --------- Co-authored-by: Andrii Shvaika <ashvayka@thingsboard.io> Co-authored-by: dashevchenko <dshevchenko@thingsboard.io>	1 year ago
..
src	EDQS (#3196)	1 year ago
pom.xml	Version set to 4.0.0-SNAPSHOT	2 years ago

Viacheslav Klimov 86b5378d59 EDQS (#3196 )

* Experiments with CSV

* CSV Loader v1

* EDQ tests

* Volatile variables instead of final

* Improvements

* updated loader with new entities

* Fix double memory usage issue

* Basic data structures and load

* Minor improvements

* Snappy + Large String reuse

* added EntityFields classes for each entity

* Basic implementation

* Minor improvements to KeyFilters

* implemented RepositoryUtils.checkKeyFilters

* Generic query implementation

* New structure

* Refactoring and few processors implementation

* extended DeviceData with shared/client attributes and device profile

* Minor refactoring of attribute scopes

* DeviceTypeFilter support

* Strong types of fields for each entity data class

* DeviceType and AssetType filters

* EntityView and Edge queries

* Relations Query

* Relation Query Implementation

* Update EDQS module version

* Sync with EDQS via Kafka

* EDQS: major refactoring

* EDQS API requests via Kafka

* EDQS: full sync with the database

* Refactoring for EDQS sync

* EDQS: major refactoring and new features

* EDQS refactoring, count query support, fix tests

* EDQS: refactoring for query processors

* Fix EDQS pom version

* Cleanup edqs.yml

* EDQS: tenant partitioning strategy; refactoring

* EDQS: latest events queue

* EDQS: support for monolith setup; RocksDB; other improvements

* EDQS: merge sync and events topics, introduce state topic

* EDQS: dynamic repartitioning

* implemented entity data query filters for edqs

* EdqsEntityQueryControllerTest - use in-memory queue

* edqs-filter fixes, added test

* EDQS: blob entity support

* EdqsEntityQueryControllerTest - use in-memory queue

* Use DummyEdqsService when disabled

* Fixes for EDQS

* Refactoring for EDQS tests

* Fix edqs requests partitioning

* EDQS: Fix for attributes handling

* Fix attributes saving in EntityServiceTest

* EDQS: refactoring, fixes

* Minor refactoring for query processor

* added ownerName/ownerType support

* fixed relation query processor

* fixed EntityServiceTest

* refactoring

* added support for parentId for relation query result

* Get rid of EntityNameFetcher

* Add fixme for relation query processor

* db restore with select all edqs fields

* fixed entity deletion

* fixed FieldUtils with new EntityFields

* dao method renamed

* EDQS: instance groups with same partitions; automatic sync; multiple fixes

* Refactoring for EDQS sync

* EDQS: refactoring

* Fix startup with Kafka

* fixed EntityQueryControllerTest

* fixed EdqsEntityServiceTest

* Separate queue admin for EDQS request template

* Implement new EDQS partitioning strategy

* EDQS: multiple fixes and refactoring

* Add mock EdqsRocksDb beans to tests

* added edqs stats for inmemory/grafana

* fixed filter tests

* Update todos

* Refactoring for QueueConfig

* Improvements and refactoring for EDQS consumers

* implemented TODOs

* test fixes

* Consume state topic up to end offsets

* edqs stats refactoring

* EDQS: cleanup on partitions removal; refactoring

* EDQS: minor refactoring

* EDQS: remove CSV loader

---------

Co-authored-by: Andrii Shvaika <ashvayka@thingsboard.io>
Co-authored-by: dashevchenko <dshevchenko@thingsboard.io>

1 year ago

src

EDQS (#3196)

1 year ago

pom.xml

Version set to 4.0.0-SNAPSHOT

2 years ago