By HusQy - 14 Oct 2018
we want to use this topic to discuss the proposal described in the series of blog posts regarding the introduction of reusable addresses in IOTA:
Part2 (Basic Idea)
Part3 (Discussion of Problems / Final Proposal)
If you have any questions or remarks regarding the proposed solution - please discuss them with us! I am looking forward to get feedback and ideas from the community.
One of the things that are particularly interesting is if the community sees a need to get this added to the protocol and what you think about some of the proposed solutions (like having to stake a certain amount of IOTA to keep a reusable address alive)
By wusyong - 14 Oct 2018
Thank you for such detailed explanation for the proposal. I have a few questions want to ask:
1. Besides transaction metadata, is there other things would change in the protocol? Like how reusable address is implemented.
Is it one address can generate multiple public keys or it's a workaround that nth public key points to Hash(seed-index)'s public key?
2. My biggest concern is "Increased storage requirements for the nodes" mentioned in part3 and solution it stated can't really convince me.
We already have wereAddressSpentFrom.txt needs to generate in snapshot files which is already controversial for its scalable problem.
Doesn't it mean that we have to create another file like this even side effect is smaller?
By projectShift - 14 Oct 2018
Just read trough the 3 blog posts, and I find it interesting that this particular subject resurfaced ... but the answer is no longer the original one, like June 2016 (I go back a while, even though you probably never heard of me, ahah)
My reading of the proposal leads me to believe that the project has now enough resources and is willing to put some effort towards providing the human side of the equation some deserved relief from the machine-focused protocol design, whilst not introducing any change that will impact it. Great!
From all you refer, the focus, imho, should be on the network validation of the transaction: we need to keep any exceptions out of the picture, IOTA needs to maintain its core ... and we can't do that if we introduce order dependency in the process or any special addresses that require a different set of rules to be processed.
If this is going to be enabled, it needs to map down to only "normal transactions" , i.e., tx that can be validated by the core IOTA protocol as designed.
With this idea in mind, I would argue that the whole solution should truly be 100% off the protocol, but also 100% in the core code, so that no node can opt-out of it and not recognize or ignore such tx - I guess that's why you're going for a protocol modification in the first place?
DNS-like solutions to keep aliases mapping to addresses sounds nice, but bring all the security issues that DNS also has (and are still to be solved today) and the additional issue of a centralized resource being in the spotlight again, like COO, so I don't think it will fly at all.
Keeping an active hash table of alias versus next key/address/whatever will bloat the store and drag down performance, plus demand higher levels of resources like memory and CPU cycles to process it continuously, so although a valid choice conceptually, I also don't think it's feasible in practice and constitutes an effective detriment to scaling.
But the mention of these two ideas did make think about a third option, that I'll layout in the most drafted of ways, without thoroughly exploring the issues it brings along right now (hopefully you experts will immediately spot them and nail the idea down and if its a nonsense, it will dye here):
What we might use to gain this functionality is to establish a service whereby a sender requests an address, by identifying the receiver by its alias, and the receiver posts an answer back in the Tangle ( a 0 tx, that doesn't truly need to be validated, but rather just read by the sender)
In order to do this securely and without the ability for a third-party to hijack the request and supply a forged answer with an address of his own, the aliases registry needs to provide a way for a 1-on-1 secure exchange ... but I would venture that such code already exists, since we already have MAM.
So, in a very rough and simplistic way as a draft, this would go like this: the address exchange would happen trough MAM , that has all the necessary mechanisms to enable it in a private and secure way, and the actual tx would be to a "normal address" from a seed controlled by the receiver, and a "normal tx" would take place, with zero exceptions and no specific code needed.
Bob's alias acts as a MAM Private channel ID (effectively rendering it public, I know, but keep up with me a little longer on this one).
Alice would post a request for an address on that channel.
Bob detects Alice's request and immediately uses channel splitting and Restricted mode to create a split channel where only he and Alice can communicate.
Bob sends Alice a regular address for the TX.
Alice uses the address given by Bob to send him the tokens and leaves the channel by posting back the tx's hash, so Bob can now follow it in the Tangle.
Bob changes the restricted key used, effectively rendering the split channel mute/closed.
Since Bob's in control all the time, only the address that he selected get's used, therefore allowing Bob to use any address he controls like he does now: one time use.
Better yet, since the address is given over a secure MAM private channel that was split with restricted mode from Bob's Private MAM channel, Bob is able to handle any number of parallel requests at the same time, by creating any number of split channels and giving a different address to each requester (or the same one, as it only matters that an address is used only once for transferring tokens out, not receiving :-) )
So Bob can be as much paranoid as he wants, by giving each requester a different address from a different seed everytime, or using the same address to concentrate the received tokens and only start using a new one after he spends from it - or any other combination in between, really, as the receiver is in total control all the time :-)
In this scenario, we do not need an alias registry at all! It's up to the receiver to post the alias he wants to use, the same all the time, and use it as channel ID of hes private MAM channel.
And no direct bloating on the store either, as all we're doing is creating more regular tx's on the Tangle, therefore enhancing the network's strength and security (although we're increasing the number of tx in the Tangle with MAM streams ... just to be able to do one more, ahah)
Ofc, I'm redirecting all of the issue towards opening and keeping a MAM channel alive, but I believe this to be fully automatable, specially in the most strong uses cases for having a permanente address for receiving, such as stores or services.
On the other hand, I may be wrong about what one can do in a MAM channel - can Alice post that initial request, or only Bob is able to post? If not, then my idea goes down the drain ... or it requires an alternative implementation of MAM - which I still think it might be a good solution for this problem.
Hope I'm not being too naive on this idea, welcome all the feedback, and also hope that I'm clearly communicating why I don't see the proposed solution as feasible.
By HusQy - 14 Oct 2018
1. No further changes are required on the "protocol" (network) level but nodes will have to adopt and take the updated public key into account when validating the signature of transactions. The way how the private and public keys for the reusable address are generated is yet to be determined, but considering that we generate the normal key pairs by doing Hash(seed + index) it would indeed make sense to generate the "other new" key pairs by just doing Hash(seed - publicKeyIndex) (especially since we propose to introduce exactly 1 reusable address per seed). This would allow us to generate a sequence of private keys that does not collide with our normal addresses.
2. wereAddressSpentFrom can be removed and will be replaced by the option to have reusable addresses. We don't have to create a new file - we already have a way to store the state of the ledger (namely the balances). It will in the worst case get 2x as big as it is today (if ALL existing addresses would be reusable). But since this scaling factor is constant, it doesnt create a problem where it scales worse than IOTA scales today already (from a complexity point of view it scales with O(n) where n is the amount of addresses that are in use [which is exactly the same as today]).
By HusQy - 15 Oct 2018
First of all let me thank your for taking the time and discussing this with us - every question and proposal will help others to clarify and understand the proposed change and also the motivations behind it.
I will try to address all of the issues and things you mentioned:
The order of transaction matters already. You can not spend from a remainder if the transaction funding this remainder was not seen / processed, yet. While it might theoretically be possible to have temporary overdrafts while doing tipselection, this is not the way IOTA currently works.
In fact most of IOTA's research department prefers to NOT allow temporary overdrafts while validating transactions and take their "history" into consideration because this will create a lot of issues of its own (see https://blog.iota.org/validity-in-the-tangle-the-good-the-bad-and-the-ugly-98bd3b53408).
The biggest one is that you can never STOP EARLY while examining tips - even if a transaction leads to a negative balance because a funding transaction MIGHT appear later. If this is not the case you are traversing unnecessary transactions with EVERY tip selection which will in return massively increase the required effort of the tip selection algorithm and degrade the tangles ability to handle a lot of transactions (because you will be busy traversing the graph in subtangles that could otherwise be judged to be invalid immediately). Taking the order of past transactions into account is therefore something that seems to be desirable from a performance point of view - even if it sometimes makes you ignore a temporarily invalid subtangle that might otherwise turn valid.
Allowing overdrafts (and ignoring the order) also prevents you to validate transactions on arrival so you can not distribute the load of the node equally over time because a transaction might suddenl turn valid at any given time.
If you look at the real world - there are a lot of "protocols" that have at least two different way to process things for different use cases (http / https, tcp / udp ... and so on) having two different ways of validating the signature of transactions based on the "use case" (humans vs machine) is not necessarily a bad thing so i disagree with you, that handling this on the protocol level is a bad design decision. Especially if implemented in the proposed way where we balance the trade-off between both approaches.
Regarding the DNS-like solution. This can easily be implemented in a decentralized way where all the nodes (that participate in this decentralized registry) have a local copy of the mappings. This way you would not face the same issues as the existing centralized hierarchical DNS infrastructure (things like cache poisoning are impossible if you update the mappings through IOTA transactions that can be validated regarding their integrity [by checking the signature and the hash]). So this would be a 2nd layer approach not affecting the performance of the protocol itself.
The solution you proposed requires the recipient to always be online OR hand over his seed to a machine that will manage the addresses for him. This is inherently insecure because this machine could be hacked. In addition i am also not convinced that we can assume that every IOTA user would set up a server that would allow him to answer to payment requests. Even if we assume that everybody would have such a machine it would allow people to spam your machine with payment requests - forcing you to generate AND monitor countless addresses (not even considering the required proof of work to answer these requests). Since IOTA is decentralized and it is hard to establish something like an identity it is also not trivial to prevent this kind of spam because you dont know who is gonna be using this service to legitimately requests a payment address to send you a donation and who just spams you.
Taking all of these aspects into account - i still think that a solution on the protocol level is the much better approach.
By projectShift - 15 Oct 2018
Thanks for the feedback, Hans!
Indeed, the biggest drawback would be the "always online" requirement, but it wouldn't matter in some use cases like stores, that already have infrastructure and services in place that can easily accommodate for one more function; you're also right when pointing out that DOS style attacks on these exposed endpoints would become a major hurdle ... but, and I realize I forgot to write it properly, in my simplistic view, the nodes would accommodate for this service - and they're already an exposed endpoint anyway - so there would never be and "offline" issue. But I did forgot to properly account for the fact that the nodes would have to handle the automated MAM channels and would require having the keys for every served alias, so it would be an impossible task, in practice.
Guess the main issue in this is that we perhaps need a working eID solution that IOTA can plug into and use for this purpose? I know work is being done on that front as well, but I would also argue for a more immediate solution to be available.
However, if you're going to do it at the protocol level, creating a first variant to support this feature, then I would at least argue for the following to be considered:
a) make it modular enough that it can later be deprecated and removed or replaced without any impact on the core scheme or any other variant
b) selecting address with index 0 as the only reusable address and 1i as the mandatory balance for use of the feature
c) make sure users can choose not to use this feature starting using addresses at index 1 and forward - wallets must implement this!
d) pay special attention to the fact that future snapshots will include a massive amount of 1i balance addresses that will never go away! Even local snapshotting will not reduce the size of this part of the Tangle! Perhaps splitting it by keeping all the index0 1i balances in a separate table? Really don't have a good idea for this, but I can sense it will become a problem
One last note regarding DNS-like solutions: even with distributed structures and IOTA assuring the integrity, you always have the issue of propagation of updated records, specially in a multi-master structure like this would be. No matter how fast you can achieve full propagation, there's no way to prevent an outdated record from being served, and thus used. That's why I don't like it for IOTA :-)
By HusQy - 15 Oct 2018
The nodes are the infrastructure for IOTA (something like the streets in the real world), they can not really take the role of managing payment requests because they are not personalized. Not every human will have his own node (especially not at the moment while we don't have mass adoption, yet).
It might happen that in the future every human will have his own node that then could do things like you described (but to be able to reach that point we have to make sure the technology is usable and appealing to the masses). So this proposal is more about "how to get there" rather than thinking about how things should be once we "arrive". Since we don't replace the current way transactions are handled this vision will not be affected by the proposed approach because you can still do all of the things you proposed (when it makes sense for the use case [like your mentioned stores example]).
I don't think it needs to ever be deprecated because it will always stay valuable at least for cluster 0 that is getting used by humans. Doing an address exchange on the tangle will increase the load on the tangle and therefore consume valuable resources that could be used for other things like value or "real" data transactions.
Since machines in the machine economy most of the time meet "face to face", they can also negotiate a new receiving address outside of the IOTA protocol within the meshnet they use to communicate (eg LIFI). So for them it will most probably not be a big issue anyway (and they also automatically have access to their seed if they have an embedded wallet). So for them your described way of communicating a new receiving address would totally be okay.
You are right about the increasing number of reusable addresses that will grow with increasing adoption but ultimately that will be covered by economic clusters. You will not have all humans on the planet use nodes with the same ledger state. I also think that being able to reuse addresses will lead to a situation where we have less addresses with higher balances. The current advice is to generate a new address for every payment so you do not run into the race conditions that were discussed in the blog posts when spending from these addresses. This means that when you receive 1000 payments you will have to keep track of 1000 addresses (on the nodes and in the wallet).
With this new approach you would only have to keep track of 1 reusable address (and the address you sweep to). This massively reduces the amount of addresses that we need to store on the nodes and decreases their storage requirements. You have to keep in mind that the additional information that we store with the address requires less space than an address itself because we don't need to store an additional balance.
So considering we had
Address1: 10 IOTA
Address2: 15 IOTA
Address3: 37 IOTA
Address4: 50 IOTA
Address5: 50 IOTA
before, we will now have
Address1 [updated public key]: 1 IOTA
Address2: 161 IOTA // address that we sweep to
Even if we spend this "sweep target" in between, we can still always use our last unspent non-reusable address as the target for newly sweeped funds. This will automatically create a situation where less addresses with higher balances need to be tracked which will reduce the storage requirements.
Another consequence of this is, that we need less "inputs" in our transactions when we send an amount that exceeds the balance of a single address, which in return means:
Since reusable addresses will allow us to get rid of the wereAddressesSpentFrom list, it will decrease the storage requirements even more.
- less transactions in the bundle
- less required proof of work
- less network traffic
- faster processing of payments in my wallet
When it comes to your idea of using index 0 as the account / reusable address - this is indeed a very reasonable thing to assume because it allows us to easily "sync" the state of the wallet between devices without having to "search" for the reusable address.
By the way: Since we can now authorize spends from the same address multiple times it is actually possible to establish something like an identity. So reusable addresses might help to implement something like an eID solution.
By agaucher - 15 Oct 2018
Let's assume that the user has several wallets (mobile and desktop version of trinity wallet by instance).
In this case, we can't have the same reusable address for both wallets.
This is annoying, and it seems complicated to explain this limitation to an ordinary user.
What is your opinion on this point ? Don't you think it could be a problem?
just another question:
What is the impact of a snapshot, from a user's point of view, when the reusable address is empty before the snapshot, but not after.
Would the case be fully managed by the wallet, or a manual action could be required by the user in this case ?
Thank you Hans,
By HusQy - 16 Oct 2018
Thank you for raising this question. You are of course going to be able to use the same reusable address for the same wallet on different devices.
Ideally we would not just store the updated public key with the address but also the public key index that was used to generate the updated public key, this will allow us to make this feature stateless and use it on different devices at the same time. Ideally you would have to use the same node for all of your devices so you don't have to worry about race conditions when you issue consecutive payments from your reusable address - but since we plan to only sweep from those addresses rather than use it for actual spends even a "race condition" would not be a problem because you would simply sweep twice, which effectively means that only one sweep would be successful.
We did not mention storing this public key index in the blog posts because it is an implementation detail that is not really necessary to understand the concept, but i can see now that we should have maybe made this a bit more clear. This public key index will not only allow the feature to be stateless but it will also allow the nodes to faster determine if a signature is valid. If the signature of the transaction that i am examining is not <currentIndex> + 1 then you can ignore the transaction and consider it invalid without even having to look at the signature because then it is a sweep that was not processed in the correct order.
This will make the processing speed as fast (or even faster) than normal transactions because we can omit the signature check).
Regarding your 2nd question: I am really not sure what you mean by this. An address will always keep the same balance when a snapshot is made. IOTA do not magically appear or disappear when a snapshot is taken
By caseyy - 16 Oct 2018
As the order of dependent transactions now matters in your reusable address proposal, you are saying it can be resolved by directly or indirectly referencing a previous spend in a consecutive spend if the previous one is still pending. I presumed, it will be considered as another tip-selection method (TSM) apart from the existing MCMC random walk tip selection algo (TSA) for choosing two tips for approving by new transaction. However, will this disrupt the original intention of Tangle MCMC of keeping the width of Tangle (see L(t) in page 7 of Tangle white paper) from either expanding too wide or shrinking too narrow?? This is because your TSM may not end-up with the similar outcome (tips selection) if you were to use TSA.
By HusQy - 16 Oct 2018
No, its exactly the same tip selection algorithm.
They way the tip selection algorithm works is by choosing a transaction in the past and then "randomly" following their approvers until we arrive at a tip. Since the walk happens from past to present, you will see the transactions in their correct order.
By caseyy - 16 Oct 2018
1). Do you have to make sure the initial location of the random walk (rw) particles are in strategic depth and position inside the Tangle such that they will pass through your previous transaction?
2). When you say the transactions are in correct order, are they in partial order or total order? If partial order, is it sufficient for your reusable address proposal?
By HusQy - 16 Oct 2018
regarding 1.) At the moment we start at a milestone that is "n" milestones in the past (relative to the latest solid milestone) so it is deterministic where we start the random walk. The depth parameter can directly be controlled in the API call "getTransactionsToApprove" (see https://iota.readme.io/reference#gettransactionstoapprove). It should be bigger than 15 to reach a good randomness. The deeper you go the more potential paths exist that you can take - so a deeper depth is usually better for the tangle.
This means you will see the transactions in the order that you attached them to the tangle. Assuming that you attached them in the correct order, you will "walk" through them in the correct order. It is of course possible that you will arrive at a later spend through the 2nd path (branch vs trunk) and therefore skip the previous spend but this "problem" also exists with the remainder approach where you might skip the transaction that funds the remainder address. In this case the 2nd spend would be considered invalid (the same way as the spend from an unprocessed remainder address would be considered invalid).
Whenever we face an invalid transaction we apply a backtracking strategy and try another path (until we find 2 valid tips).
regarding 2.) Since it is possible that we "skip" transactions (due to the fact that there are usually 2 path that exist to arrive at any given point), they are only in partial order.
If you would set the "trunk" to be the same transaction as the "branch", you could prevent the possibility of reaching the 2nd spend without having walked through the first one but that would also decrease the chances of your transaction to be reached.
Since we plan to only sweep the balances every now and then (wallets could implement sth like only sweep if your last sweep is not older than n minutes) this is something that can be worked around to be practically irrelevant.
By caseyy - 17 Oct 2018
Setting the "trunk" (Tip 0)to be the same transaction as the "branch" (Tip 1), are you saying only one tip to approve, that it will end up like ''chain'' instead of DAG? That is why it is undesirable?
I appologize if my questions are naive. I understand the inherent partial order of DAG. Let say, if DAG were able to be total ordered(perhaps using modified DFS) first and logically timestamped, and the TSA were to check this ordering timestamping also(on top of checking the accumulated weight) , will it helps?
By HusQy - 17 Oct 2018
Yes exactly, you would only choose 1 Tip in this case and this one transaction would then indeed be "chained" to the previous one. This is possible already and we see it happen in the tangle, today.
But as i already said, this slightly decreases the chance of your transaction to be picked up by other nodes doing tip selection because there exists only 1 path to reach that chained transaction (instead of 2). It is not necessarily bad for the tangle because other people would still reference 2 tips. Since everybody wants their transactions to be picked up by other nodes it is in everybody interest to reference two other transactions. In math terms you call this a nash equilibirum - a rule that (even tho it doesnt get enforced) will be followed my most participants voluntarily because it is in their very own interest.
It would maybe even make sense to "chain" consecutive sweeps together this way because if the 2nd path to my transaction skips the first sweep - this path would always consider my sweep invalid anyway. And having an invalid path is as good as having just 1 anyway. Ideally we would select two tips that both indirectly reference my first spend. This can be done by starting the random walk at my first sweep instead of a milestone in the past. This can be done by providing the "reference" parameter in getTransactionToApprove.
Btw. these are very important things to ask and understand - the questions are not naive at all.
If the DAG could somehow be totally ordered this would solve A LOT of problems but since nodes and transaction creators can lie about the timestamps it is not easy (maybe even impossible) to achieve
By caseyy - 31 Oct 2018
Does IOTA Tangle use physical clocks or logical clock for its timestamping mechanism? Thanks.