[Question][Question]Full The full nodes of Tron have been stuck at block

[letmountain]

The same problem occurred again, this time stuck at 81346296 without any power outage, resulting in abnormal shutdown

tron.log.zip

[letmountain]

https://github.com/tronprotocol/java-tron/issues/6910

[halibobo1205]

What's happening

Your local database is corrupted — its stores are left in an inconsistent state, and the node is now permanently deadlocked.

Your properties store records the head as 81345991, but the block store on disk already contains blocks 81345992–81346296 (305 blocks). The node therefore only ever requests 81346297, whose parent block is not in the in-memory khaosDb (khaosDb is seeded only from the head at startup). That throws UnLinkedBlockException — the exception carries no message, which is why the log shows reason: null — the block is treated as a bad block, the peers are disconnected, and the cycle repeats forever.

How to recover

The dataset has to be replaced with a consistent one.

Official snapshots: https://developers.tron.network/docs/main-net-database-snapshots#choosing-a-snapshot

What we need from you

This is the second occurrence, so unless we find the cause it may well happen again after you re-download.

1. Config — the storage section, specifically: snapshot.maxFlushCount, checkpoint.version, checkpoint.sync, db.sync, db.engine (just say "default" for any you haven't set).

2. The log from the day it broke — no need to guess the date; once the node is stuck it stops flushing entirely, so you can find it by content:

grep -l "Flush cost" /path/to/logs/tron.log* | sort | tail -3

The last file that matches is the last day the node was writing normally — the next one after it is the incident. Please attach both.

3. Where the current data came from — which snapshot did you download, and from where? Full or lite? Was it synced by this node, or copied/restored from somewhere? Also please run:

ls -l --time-style=full-iso /path/to/output-directory/database/checkpoint/

The directory names are millisecond timestamps, so this pins down exactly when the node last flushed to disk.

4. What operations were performed — around the time it broke: version upgrades, config changes, backups, copying the database directory, any Toolkit commands, host migration, and so on. In particular: when the database directory was copied, was the node stopped or still running? And were both occurrences on the same machine?

[letmountain]

@halibobo1205 1、

2、

tron-2026-08-26.0.log.zip

3、3197828606537 Feb 4 2026 FullNode_output-directory.tgz,The current data was extracted from a snapshot taken on February 4th

4、No operations were performed, and the monitoring indicates that the block is out of sync

[halibobo1205]

@letmountain

Thanks for the additional log. This changes one point in our earlier assessment: the exit at 06:32 on August 26 was not a normal shutdown.

While processing block 81346317 and creating a checkpoint, LevelDB failed to create its temporary file:

checkpoint/1787725944800/000002.dbtmp: No such file or directory

The process then exited with LEVELDB_INIT(1). The subsequent close end only shows that the shutdown hook completed its cleanup; it does not indicate that the database or storage was healthy at that time.

During shutdown, TxCacheDB recorded currentBlockNum: 81346296. This value is read through getLatestBlockHeaderNumberFromDB() and therefore comes from the persisted root store. In the current database, properties is at 81345991, while block still contains blocks 81345992–81346296.

Therefore, assuming both runs used the same data directory, the rollback from 81346296 to 81345991 was persisted after the 06:32 shutdown. However, the underlying cause may already have been present before then—the checkpoint directory failure at 06:32 could have been the first visible symptom of the same problem.

A disk or filesystem issue is worth investigating. However, the reported error is No such file or directory, rather than an explicit disk-full, read-only, or hardware I/O error. It indicates that the path did not exist or was temporarily unavailable when LevelDB attempted to create the file. Possible causes include concurrent directory deletion, a mount issue, a filesystem problem, or checkpoint cleanup.

To narrow this down, could you provide the following information?

1. Log from the first restart after the 06:32 exit

Please include everything from process startup through the following lines, preserving the original ordering, timestamps, and surrounding context:

checkpoint version: X
checkpoint sync: X
checkpoint v2 recover success

or:

checkpoint size is 0, using v1 recover

and:

Current headNum is: X

Current headNum is printed after checkpoint recovery completes. If it is already below 81346296, that confirms the rollback had occurred by the end of the first restart’s recovery process. The surrounding recovery logs should help us narrow down when it was applied.

2. System and disk logs

First, please provide the complete kernel log around the 06:32 failure:

journalctl -k \
  --since '2026-08-26 06:20' \
  --until '2026-08-26 06:45' \
  --no-pager

Then run a filtered check over the entire day. The known problem window is from 06:32 to 14:25, so checking only the immediate failure period is not sufficient:

journalctl -k \
  --since '2026-08-26' \
  --until '2026-08-27' \
  --no-pager |
grep -iE 'I/O error|EXT4-fs|XFS|blk_update_request|Buffer I/O|read-only|medium error|reset|timeout'

Please also provide the filesystem and mount information:

df -hT /data/tron
df -i /data/tron
findmnt -T /data/tron/output-directory/database

If this is a bare-metal host, please also check the SMART data for the underlying disk:

smartctl -a /dev/<device>

3. Checkpoint directory and cleanup jobs

If the old checkpoint directory still exists, please do not delete it. Please provide:

ls -ld /data/tron/output-directory/database/checkpoint
ls -la --time-style=full-iso \
  /data/tron/output-directory/database/checkpoint

Also, please confirm whether any cron job, startup script, operations script, or other process might clean up, move, or recreate this directory.

4. Restart mechanism and data directory

How was the node restarted after the 06:32 exit—systemd, Docker, supervisor, Kubernetes, a scheduled script, or manually?

Please also provide the actual startup command, configuration file path, and data directory so we can confirm that both runs used the same database directory.

5. Time synchronization records

A system clock change is only one possibility we want to rule out; it is not a confirmed cause. Please provide:

journalctl --utc -o short-iso-precise \
  -u chronyd -u chrony -u systemd-timesyncd -u ntpd \
  --since '2026-08-25' \
  --until '2026-09-01' \
  --no-pager

timedatectl

If this is a virtual machine, please also check whether the host performed a snapshot restore, suspend/resume, live migration, host time correction, or forced guest time synchronization around August 26.

---

Re-downloading a snapshot can restore the node for now. However, since this is the second occurrence, we recommend keeping the logs and the old database directory until the root cause is identified, to reduce the risk of the same problem affecting the new database.

[letmountain]

@halibobo1205 1、

tron-2026-08-27.0.7z.zip

2、

manually

4、

[halibobo1205]

Root cause found: the host clock jumped, twice. @letmountain

On 2026-08-26 the system clock moved forward by 7h56m45s and then back again ~30 seconds later:

06:28:24.808  Receive BLOCK Num:81346349
14:25:10.113  Process transaction ... cost 28605274 ms during apply      ← = 7h56m45s, the jump itself
14:25:40.665  Receive BLOCK Num:81346506
06:28:55.432  ...                                                        ← jumped back

Confirmed on the host side too: systemd-journald: Time jumped backwards, rotating. and systemd-timesyncd: Initial clock synchronization to Wed 2026-08-26 06:28:55.

What to do

1. Re-download the snapshot (you already did) — that recovers the node.

2. Worth investigating why the clock moved by that much. Time synchronisation itself is of course normal, but a routine correction is milliseconds and is slewed rather than stepped. Here the clock first moved ~8 hours forward at 06:28:24, and systemd-timesyncd's initial synchronisation then stepped it back — so the forward move is the anomaly. Two things that may help narrow it down:

   • The size, 7h56m45s, is within ~3 minutes of the UTC+8 offset, which often indicates a clock being read as local time instead of UTC somewhere in the chain.
   • This is a VMware guest, so it is worth checking whether VMware Tools time sync and systemd-timesyncd/chrony are both active, and whether the host's clock/timezone is correct. A VM snapshot restore or resume can produce the same effect.

   It would also help to start java-tron only after time sync has settled (After=time-sync.target / Wants=time-sync.target). Note there is another Initial clock synchronization on 08-28 06:35, so the pattern does recur, which matches this having happened twice.

We're still analysing the checkpoint V2 code path and will follow up with concrete hardening suggestions.

Thanks for providing such detailed logs and system information — that's what made the root cause findable.