Ecosystem chain is not finalized. How to recover from this?

Hi, we are an ecosystem team and has been running our testnet with external nodeOps teams. Recently we encounter a situation that a block does not finalize and we have no clue why.

Here is the extracted log:

The block with problem is block #107380 (line 1). Following the log, there are some reorg around #107379 (line 6, 11, 21), and then the finalized block never proceed to #107380, and eventually there is a block import error (line 123).

If you are interested in the transactions included, here is for #107380.

My questions are:

  1. Is there any clue to know what is causing the non-finalization issue?
  2. Without resetting the chain (even though this is a testnet, we have partners running the chain together), how can we make the chain finalize and proceed forward again?

Thank you!

Hello. Seems like your last produced block was also one hour back which is expected since if your finalisation is halted gradually your block production time also increases significantly. One possible reasons could be you have 4 validators in the active set. check their disk spaces first. if disk spaces got filled out there will be DB read and write problems which may cause this issue.

1 Like