fix: Harden metrics and event subsystem under stress by nikooo777 · Pull Request #720 · permaweb/HyperBEAM

nikooo777 · 2026-03-02T18:22:11Z

Actions

Update production config to use "only-ids": true option in remote stores.

Tests

rebar3 eunit --module dev_node_process still fails on 2 tests, but this is happening in edge as well.

Summary

Fix stale PID caching in event server causing frozen metrics in long-lived processes
Add timeout to cpu_sup:avg5 scrape to prevent indefinite blocking
Use selective ETS query for /events endpoint instead of full table scan
Remove unnecessary spawn-per-request in metric recording paths
Guard all unprotected Prometheus writes that could crash critical server processes
Cache Prometheus startup failure with 60s retry backoff to avoid hot-path blocking
Bound event server Prometheus wait to ~30s with lazy counter redeclaration

Commits

3fc9e78 fix: Harden event server against crash and stale PID caching
e872f1f fix: Add timeout to cpu_sup:avg5 in metrics collector
6e1a978 fix: Use selective ETS query for event counters
da2c207 fix: Remove unnecessary spawn in metric recording paths
1c25e27 fix: Guard unprotected Prometheus writes in hot paths
c0b8005 fix: Cache Prometheus startup failure with retry backoff
535c2c6 fix: Bound event server Prometheus wait and handle late startup

src/hb_http_client.erl

src/hb_http.erl

speeddragon · 2026-03-06T19:54:02Z

hb_store_gateway: remote_hyperbeam_node_ans104_test test is fixed in #736

speeddragon · 2026-03-07T00:20:40Z

Only tests now failing are:

hb_client (5 failing)
dev_arweave (2 failing)

Due to how the upload is done to up.arweave.net

…st-hook" This reverts commit de6262b, reversing changes made to 37c508f.

- Wrap prometheus_counter:inc/3 in try-catch in handle_events/0 to prevent server crash on Prometheus errors - Add is_process_alive/1 check in find_event_server/0 so long-lived processes recover from a dead cached PID

- Wrap cpu_sup:avg5/0 in spawn_monitor with 2s timeout - Kill worker on timeout to prevent process leaks

- Replace ets:tab2list with ets:match_object filtering for event rows - Avoids materializing the entire prometheus_counter_table

- Inline Prometheus ETS writes in hb_http and hb_store_arweave - Guard observe in hb_http_client record_duration spawn - Eliminates per-request process overhead under load

- Add try-catch to dec_prometheus_gauge and inc_prometheus_counter - Guard retry path in inc_prometheus_gauge - Wrap hb_store_lmdb:name_hit_metrics to prevent LMDB read crashes

- Check ensure_all_started result instead of ignoring it - Cache failure timestamp in persistent_term, retry after 60s - Callers degrade gracefully when Prometheus is unavailable

- Cap await_prometheus_started to ~30s to prevent unbounded mailbox growth - Messages stay in mailbox during wait instead of being consumed and lost - Wrap declare in try-catch, redeclare lazily on inc failure via ensure_event_counter

- Keep metric writes off the HTTP reply path to isolate request tail latency - Retain try-catch inside the spawn to guard against metric-path hiccups

…store_arweave

…est-hook" This reverts commit 9f0d114.

speeddragon · 2026-03-10T15:12:05Z

src/hb_event.erl

+    ets:match_object(
+        prometheus_counter_table,
+        {{default, <<"event">>, '_', '_'}, '_', '_'}
+    ).


Instead of returning all items from the list, we only return events.

Does this materially effect the results on ~hyperbuddy@1.0/events/format~hyperbuddy@1.0?

No. This filters from the ETS side, before counters, which also filters by <<"event">> only counters. This just reduces the amount of data ETS sends to this process, which would be discarded further down.

speeddragon · 2026-03-10T15:13:28Z

src/hb_http.erl

            }
    end,
-    WithPrivIP = hb_private:set(NormalBody, <<"ip">>, RealIP, Opts),
+    WithPrivIP = hb_private:set(WithPeer, <<"ip">>, RealIP, Opts),


WithPeer wasn't being used.

src/hb_http.erl

…allet

src/dev_name.erl

speeddragon · 2026-03-10T18:54:50Z

src/dev_bundler.erl

        NodeOpts2 = maps:merge(NodeOpts, #{ bundler_max_items => 3 }),
        Node = hb_http_server:start_node(NodeOpts2#{
-            priv_wallet => hb:wallet(),
+            priv_wallet => ar_wallet:new(),


Since we now start hb for every single test, we cannot use hb:wallet because the ServerID address will conflict when starting a new node inside the test. By using ar_wallet:new(), a new ServerID will be generated based in the private wallet, and it will not conflict with existing one.

This is because hb:wallet() only creates a wallet if one hasn't been cached in the persistent_term store?

I think it uses the priv_key_location, which defaults to <<"hyperbeam-key.json">>.

samcamwilliams · 2026-03-10T19:04:05Z

src/hb_ao_test_vectors.erl

+    % There is a race condition where we write to the store and a 
+    % reset happens making not read the written value.
+    % Lower this number can still produce flaky test
+    timer:sleep(100),


In general it is better to use hb_util:until for this type of thing.

I can't find something to wait for. Maybe I can use hb_store:list and make sure it is empty, but when I tried that it got stuck there (not sure why).

samcamwilliams · 2026-03-10T19:06:24Z

src/hb_http.erl

        hb_http_client:response_status_to_atom(Status),
        http_response_to_httpsig(Status, Headers, Body, Opts)
+    };
+outbound_result_to_message(<<"json@1.0">>, Status, Headers, Body, Opts) ->


Why does this need a custom handler? This could just match outbound_result_to_message(Codec, Status, Headers, Body, Opts) and convert to the given codec.

samcamwilliams · 2026-03-10T19:11:31Z

src/hb_prometheus.erl

-                [prometheus, prometheus_cowboy, prometheus_ranch]
-            ),
-            ok;
+            case persistent_term:get(hb_prometheus_start_failed, undefined) of


We definitely don't want this route. I don't think it works and it may also return non-ok, when it is called ensure_. Why not 38ccd088ff4cdd1c79bfeb09fd88fde0bb7e0d6e? (Also, we should make sure 14827ad3ee2a3f53803f795cc6ce250794e1240a or equivalent makes it in from this dead branch).

Applied both commits.

samcamwilliams · 2026-03-10T19:12:32Z

src/hb_store_gateway.erl


 %% @doc Reading an unsupported signature type transaction should fail
-failure_to_process_message_test() ->
+%% TODO: Enable when we find a TX that we don't support


Low-key flex 😂

speeddragon · 2026-03-11T19:50:56Z

src/dev_name.erl

                <<"host">> => <<"draft-17_whitepaper">>
            },
-            Opts
+            Opts#{http_client => httpc}


Temporary fix until we fix this in gun.

speeddragon reviewed Mar 2, 2026

View reviewed changes

src/hb_http_client.erl Outdated Show resolved Hide resolved

src/hb_http_client.erl Outdated Show resolved Hide resolved

src/hb_http.erl Show resolved Hide resolved

nikooo777 force-pushed the fix/neo-logging branch from 535c2c6 to 0fb60f4 Compare March 3, 2026 03:38

speeddragon force-pushed the fix/neo-logging branch from 0fb60f4 to 6dde746 Compare March 5, 2026 14:12

speeddragon changed the base branch from neo/edge to impr/gun March 5, 2026 14:12

speeddragon force-pushed the impr/gun branch from fef736c to 4752b18 Compare March 5, 2026 16:48

speeddragon force-pushed the fix/neo-logging branch from 872e966 to 5d84e2f Compare March 7, 2026 00:17

samcamwilliams and others added 15 commits March 9, 2026 18:26

Revert "Merge pull request #729 from permaweb/speeddragon/impr/manife…

9f0d114

…st-hook" This reverts commit de6262b, reversing changes made to 37c508f.

fix: Harden event server against crash and stale PID caching

d34e9d0

- Wrap prometheus_counter:inc/3 in try-catch in handle_events/0 to prevent server crash on Prometheus errors - Add is_process_alive/1 check in find_event_server/0 so long-lived processes recover from a dead cached PID

fix: Add timeout to cpu_sup:avg5 in metrics collector

3c2fea1

- Wrap cpu_sup:avg5/0 in spawn_monitor with 2s timeout - Kill worker on timeout to prevent process leaks

fix: Use selective ETS query for event counters

cb420a7

- Replace ets:tab2list with ets:match_object filtering for event rows - Avoids materializing the entire prometheus_counter_table

fix: Remove unnecessary spawn in metric recording paths

d1cf71f

- Inline Prometheus ETS writes in hb_http and hb_store_arweave - Guard observe in hb_http_client record_duration spawn - Eliminates per-request process overhead under load

fix: Guard unprotected Prometheus writes in hot paths

e7a759e

- Add try-catch to dec_prometheus_gauge and inc_prometheus_counter - Guard retry path in inc_prometheus_gauge - Wrap hb_store_lmdb:name_hit_metrics to prevent LMDB read crashes

fix: Cache Prometheus startup failure with retry backoff

f0eaf71

- Check ensure_all_started result instead of ignoring it - Cache failure timestamp in persistent_term, retry after 60s - Callers degrade gracefully when Prometheus is unavailable

fix: Restore spawn in record_request_metric for latency isolation

21664be

- Keep metric writes off the HTTP reply path to isolate request tail latency - Retain try-catch inside the spawn to guard against metric-path hiccups

fix: dev_copycat_arweave test

b2ff147

disable failure test, until we find a unsupported tx

dd0c3e4

fix: Test in dev_codec_tx, hb_ao_test_vectors, hb_http_client and hb_…

ffed499

…store_arweave

fix: Add host to httpsig requests

144c6b5

fix: prometheus declare instead of new

d3a98a5

fix: init hb on hb_client

33a7cbd

speeddragon force-pushed the fix/neo-logging branch from 4ffac84 to 33a7cbd Compare March 9, 2026 18:27

speeddragon added 2 commits March 9, 2026 18:52

Reapply "Merge pull request #729 from permaweb/speeddragon/impr/manif…

2aefce0

…est-hook" This reverts commit 9f0d114.

impr: prometheus handling and fix POST tx endpoint

1dca459

speeddragon force-pushed the fix/neo-logging branch from dc33e0c to 1dca459 Compare March 10, 2026 14:40

speeddragon added 2 commits March 10, 2026 14:48

impr: use singleton pattern

ff73250

fix: increase timer on as_load test to temporarly fix test

de2c82c

speeddragon reviewed Mar 10, 2026

View reviewed changes

src/hb_http.erl Outdated Show resolved Hide resolved

speeddragon added 3 commits March 10, 2026 16:07

fix: dev_name test that was broken by providing host header now

e4cfc0e

impr: Add option to remote store to only read IDs

ece67f9

fix: dev_blacklist test when starting a new node with existing priv_w…

79a4f41

…allet

speeddragon reviewed Mar 10, 2026

View reviewed changes

src/dev_name.erl Outdated Show resolved Hide resolved

speeddragon added 2 commits March 10, 2026 18:39

fix: dev_bundler node start, when we already started one

e63b8c1

impr: Testing documentation

1339453

speeddragon marked this pull request as ready for review March 10, 2026 18:44

speeddragon reviewed Mar 10, 2026

View reviewed changes

Update hacking-on-hyperbeam.md

46114ab

samcamwilliams reviewed Mar 10, 2026

View reviewed changes

speeddragon added 3 commits March 11, 2026 17:56

fix: tests with wallet

93b924c

fix: Make dev_rate_limit use singleton

ed5a7c8

fix: Revert changes, use httpc instead of gun in arns test

7f04020

speeddragon force-pushed the fix/neo-logging branch from fb25fcd to 7f04020 Compare March 11, 2026 19:15

samcamwilliams added 2 commits March 11, 2026 19:35

impr: gun event logging

ccd23ca

impr: prometheus management

3f26d45

speeddragon reviewed Mar 11, 2026

View reviewed changes

fix: Remove comment

a536eca

Conversation

nikooo777 commented Mar 2, 2026 • edited by speeddragon Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Actions

Tests

Summary

Commits

Uh oh!

Uh oh!

Uh oh!

Uh oh!

speeddragon commented Mar 6, 2026

Uh oh!

speeddragon commented Mar 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

speeddragon Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nikooo777 commented Mar 2, 2026 •

edited by speeddragon

Loading

speeddragon Mar 10, 2026 •

edited

Loading