Two related fixes for a stuck-open 'Monitoring' entry in the incident header:
1. serviceIsOk(GENERAL) is now called when a monitoring cycle completes
successfully. Previously GENERAL could only accumulate failures (via the
outer Throwable catch), with no complementary recovery, so once the
catch-all fired the service stayed red forever.
2. checkEdqs() is now wrapped in its own try/catch that reports any
non-ServiceFailureException failures under EDQS rather than GENERAL.
Connection/read timeouts hitting /api/entitiesQuery/find previously
propagated unwrapped and were bucketed as GENERAL, which hid the fact
that EDQS was the failing component.
The resolution countdown now only starts once every failing service has
recovered. While any service is still firing the timer is cancelled, so
the incident cannot auto-resolve and spawn a new incident for the same
ongoing failure between alerts.
High latency is a warning signal (no explicit recovery event) and
therefore does not block resolution — only FAILING services do.
Spring wraps Apache HttpClient connect failures as 'I/O error on POST request:
Connect to <URL> failed: <reason>'. Same treatment as the RestClient variant:
the URL is embedded in the 'request' word and the 'Connect to ... failed'
restatement is dropped from the visible text.
Recovered services are now shown as ':large_green_circle: <name> (<count>)'
with the last failure count that was observed before the service recovered,
matching the red-circle format.
Transforms 'request for "<URL>"' into Slack mrkdwn <URL|request>, so the
failure line renders the word 'request' as a clickable link instead of
showing the raw URL in quotes.
Changelog review (v1.40..v1.48): no breaking changes to the slack-api-client
APIs used in this codebase. The only breaking changes in v1.45.0 were in
slack-app-backend (servlet classes) and slack-api-bolt-aws-lambda-s3-storage
(AWS SDK v1→v2), neither of which we depend on. Transitive deps (okhttp,
gson, okio) are unchanged.
Verified by compiling monitoring + application against 1.48.0 and running
the monitoring unit tests plus the application Notification/Slack test
suites (15 + 54 tests, all pass).
Adds Slack-API-based incident grouping for the monitoring microservice:
alerts that fire within resolution_timeout_s are threaded under a single
"Incident" message whose header tracks affected services in real time, and
the incident auto-resolves after a quiet period with a final summary.
Highlights
- Dual Slack modes: when bot_token + channel_id are set, alerts go through
chat.postMessage / chat.update with threaded replies; otherwise the
existing webhook path is used unchanged.
- Incident header shows 🔴 failing / :large_yellow_circle: high
latency / :large_green_circle: recovered services with live failure
counts and elapsed duration (updated every minute).
- Notifications carry structured affected-service data
(AffectedService { name, status, failureCount }) so the incident layer no
longer parses formatted alert text with regex.
- IncidentManager is decoupled from Slack via a small IncidentTransport
interface; SlackIncidentTransport adapts SlackApiClient.
- PE/other service-key types can plug in a friendly name via the
ShortNameProvider interface; TransportInfo implements it.
- Config lives under monitoring.notifications.incident.* (enabled,
resolution_timeout_s, tag_channel). Slack bot config stays under
monitoring.notifications.slack.{bot_token,channel_id}. YAML defaults are
authoritative; Spring @Value no longer carries a conflicting fallback.
- Concurrency: state and transport I/O run under the manager's monitor.
Slack client has explicit 5s call timeouts (set on SlackConfig) so the
hold time is bounded. Slack client is closed on PreDestroy.
- HTTP failure text is sanitised: HTML response bodies are stripped so
Nginx-style error pages don't flood alerts.
- BaseMonitoringService splits login / WS connect / WS subscribe into
distinct MonitoredServiceKey entries, uses catch(Exception) instead of
catch(Throwable), and wraps WsClient in try-with-resources.
- Unit tests cover incident lifecycle, status transitions, duration
formatting, HTML body stripping, and the ShortNameProvider dispatch.
TbRestApiCallNodeTest ran concurrently with SsrfSafeAddressResolverGroupTest,
which toggles the static SsrfProtectionValidator.enabled flag in its
setUp/tearDown. When the flag leaked into the REST test's async HTTP calls,
'localhost' was rejected by SSRF and extra tellFailure invocations broke the
Mockito verify count.
TbHttpClientTest and SsrfSafeAddressResolverGroupTest already declare
@ResourceLock("SsrfProtectionValidator"); apply the same lock to
TbRestApiCallNodeTest so all three SSRF-sensitive tests serialize.
Fixes#15453
Prevents UnrecognizedPropertyException during rolling upgrades when a
newer node writes a cached entity with an added field and an older node
reads it back. The Redis-backed TbJsonRedisSerializer now uses
JacksonUtil.IGNORE_UNKNOWN_PROPERTIES_JSON_MAPPER instead of the strict
OBJECT_MAPPER used by JacksonUtil.fromBytes.
Relax version check to allow maintenance digit increases within the same
LTS family (e.g. 4.3.0 -> 4.3.1), not just patch digit increases.
Add LTS SQL schema patch execution from upgrade/lts/schema_update.sql,
running before views and widget updates so schema changes are in place
for dependent objects.
Replace the blocking semaphore guard with a non-blocking bounded FIFO queue
+ semaphore pattern:
- No semaphore/queue when maxParallelRequestsCount=0 (default): direct doHttpCall,
identical to the old behavior.
- When a concurrency limit is set, incoming messages are enqueued via non-blocking
offer(); a full queue triggers onFailure immediately.
- tryProcess() acquires one semaphore slot and dispatches the next valid queued task.
Stale tasks (batch deadline expired) are dropped and the slot reused in the same pass.
- doFinally hook releases the semaphore and calls tryProcess() exactly once after any
terminal signal (success, error, cancel), preventing double-release and permit leaks.
- publishOn(externalCallExecutor) moves callbacks off reactor-netty I/O threads.
System-level safety caps are wired through thingsboard.yml → ActorSystemContext →
TbContext → TbHttpClient, scoped to rule-engine services only via @TbRuleEngineComponent:
actors.rule.external.http_client.max_parallel_requests (ACTORS_RULE_EXTERNAL_HTTP_CLIENT_MAX_PARALLEL_REQUESTS)
actors.rule.external.http_client.max_pending_requests (ACTORS_RULE_EXTERNAL_HTTP_CLIENT_MAX_PENDING_REQUESTS)
actors.rule.external.http_client.pool_max_connections (ACTORS_RULE_EXTERNAL_HTTP_CLIENT_POOL_MAX_CONNECTIONS)
Backward compat: TB_RE_HTTP_CLIENT_POOL_MAX_CONNECTIONS still honored via yaml fallback.
Observability: five AtomicLong counters (dispatched, success, failure, droppedQueueFull,
droppedStale) with periodic WARN anomaly logging including semaphorePermits for leak detection.
No configuration changes or upgrade scripts required — docker image update is sufficient.
Rename RestApiCallNodeSettings to TbHttpClientSettings
The settings are about HTTP client transport concerns (connection pool,
concurrency, queue depth), not REST API Call node business logic.
The new name matches the consumer (TbHttpClient) and the YAML path
(actors.rule.external.http_client.*).
Oleksii Kuripko
Viacheslav Klimov
Sergey Matvienko
Vladyslav_Prykhodko
Maksym Tsymbarov
Oleksandra Matviienko