Hello IOTA Forum

LatestSolidSubtangleMilestoneIndex suddenly jumps up to a high value

https://forum.helloiota.com/Topic6495.aspx

By StefanEkstam - 23 Dec 2017

I have tried getting my full IOTA node working for a couple of weeks now. Yesterday I updated to IRI 1.4.1.4, and some of the problems were apparently addressed in the new version.

However, my main problem remains: After stopping and restarting the node a couple of times, the LatestSolidSubtangleMilestoneIndex jumps up to a value quite near the LatestMilestoneIndex, although it is actually nowere near it in reality.

A description of the process:
- I start the node with an empty mainnetdb directory
- IRI connects to my neighbors
- Fairly immediately, IRI says "Latest milestone has changed from #243000 to #309994"
- After a short while, IRI says "Latest SOLID SUBTANGLE milestone has changed from #243000 to #243006"

Then I let the node run, and everything seems to work just fine. Both the latest milestone and the lateset solid subtangle milestone keep updating.

If I now stop IRI (with Ctrl-C) for some reason, it shuts down nicely, without any error messages. However, when it comes up again, I no longer receive any latest solid subtangle milestone updates. If i perform a getNodeInfo call to the node, it says that my latest solid subtangle milestone index is 309699, although it should in reality probably be somewere near 244000. Then the index is stuck at 309699 and never updates again.

So it seems that the latest solid subtangle milestone index for some reason gets corrupted when shutting down.

Is there any way of fixing this problem, apart from starting all over again?

Regards,
Stefan

By tawago - 23 Dec 2017

@StefanEkstam
There is a known issue for the sub milestones on 1.4.1.4 that a lot of people are having hard time with.
There will be a next version update in coming future, i think.
https://github.com/iotaledger/iri/pull/435
By StefanEkstam - 24 Dec 2017

tawago - 23 Dec 2017
@StefanEkstam
There is a known issue for the sub milestones on 1.4.1.4 that a lot of people are having hard time with.
There will be a next version update in coming future, i think.
https://github.com/iotaledger/iri/pull/435

Hello tawago,

Thanks for the link!

It's actually quite scary that this kind of severe bugs still exist in the code, imo. I have invested many days in this now, trying to get my full node to work. Now I know that it's really no idea to continue, since I will never be able to fully catch up, the way the node code works right now.

I'll probably just let my node run, communicating with my neighbors, but without any hope to get fully synced until the next version is released.

Oh well, life is harsh... Smile

Regards,
Stefan
By thorium - 27 Dec 2017

I was having similar issues. This is what worked for me:

Download latest snapshot from http://db.iota.partners/IOTA.partners-mainnetdb.tar.gz (updated every 30 minutes)

Stop iri daemon and delete content in mainnetdb folder and mainnetdb.log folder. 

Extract the new snapshot inside mainnetdb folder. 

Check directory permissions in case you changed anything.

Start iri as normal (no parameters)

It survived 3 restarts so far.
By thorium - 27 Dec 2017

...and it´s stuck again
By timeofmind - 28 Dec 2017

I have the same issue here. I've had the full node going for weeks but can never get fully sync'd. I've used all techniques published on the net, including downloading the latest mainnetdb and restarting the node, which I've done a few times but with no success of ever reaching the latest milestone.
By thorium - 28 Dec 2017

Had to do a restart yesterday and still going strong. Using Nelson btw to fill up the server mixed with some static nodes. It should sync quite quickly (a few hours) once the newest snapshot is installed.
By StefanEkstam - 31 Dec 2017

I've been trying for weeks now, on UNIX machines and on Windows machines, both physical and virtual. Up till yesterday I had a fresh server running for five days in a row, incrementing the solid subtangle index from 243000 to around 289000, and then it crashed.

As always, when restarting the node, the solid subtangle index was set to just below the latest milestone index, from where it will never again change.

Performing a rescan doesn't help.

Downloading the latest mainnetdb doesn't help either, because as soon as the node crashes, it will invalidate the solid subtangle index.

This is just sad
By StefanEkstam - 31 Dec 2017

After running a while, my node now reports the following:

12/31 02:49:18.060 [Solid Milestone Tracker] INFO com.iota.iri.Snapshot - Transaction resolves to incorrect ledger balance: -460066574127018

Well, at least it's something new and interesting happening. Not that it helps...
By timeofmind - 31 Dec 2017

Yesterday my node was at a latestMilestoneIndex that matched the latest milestone on major public nodes, but the latestSolidMilestoneIndex was stuck. Today the latestMilestoneIndex is stuck at 293770, which is not unusual, I've seen this many times before... and the latestSolidMilestoneIndex is moving ahead but way behind, at 267140.

I can see the node just crashed 2 hours ago with:

Dec 30 20:25:24 vps163888 java[10030]: terminate called after throwing an instance of 'std::bad_alloc'
Dec 30 20:25:24 vps163888 java[10030]: what(): std::bad_alloc
Dec 30 20:25:25 vps163888 systemd[1]: iota.service: Main process exited, code=killed, status=6/ABRT
Dec 30 20:25:25 vps163888 systemd[1]: iota.service: Unit entered failed state.

Which is also typical... happens all the time. My vps has plenty of memory... it is not running out of memory. It has 8GB total and 5GB are allocated to the java virtual machine.
By timeofmind - 31 Dec 2017

Even with heap size set at 5GB  ( -Xmx5G ), the java process was using close to 7GB. I've decreased to -Xmx4G and will see how this does. I have 8GB of memory total.
By timeofmind - 31 Dec 2017

This has got to be a bug. Even limiting the jvm heap size to 4GB (-Xmx4G) on an 8GB system leads to this memory error and the iota node crashing. Perhaps a core dump will tell me something?

Edit: The only thing I'm running other than the IOTA node is Nelson.
By timeofmind - 31 Dec 2017

I just noticed these logs with iota iri running under debug:

2017/12/31-00:43:25.000072 7fba9affd700 (Original Log Time 2017/12/31-00:43:24.996061) [db/memtable_list.cc:372] [bundle] Level-0 commit table #160221 started
2017/12/31-00:43:25.000075 7fba9affd700 (Original Log Time 2017/12/31-00:43:25.000002) [db/memtable_list.cc:395] [bundle] Level-0 commit table #160221: memtable #1 done

Clearly the java heap size has little to do with how much memory the iota node utilizes, as it is making use of c libraries. Clearly 8GB is just not enough memory to run a full IOTA node. Hopefully 12GB will be enough. I'll have to upgrade to a more powerful VPS yet again...

By thorium - 31 Dec 2017

Stefan: any clues to why it is crashing? Both windows and unix?