Ah, okay, thanks it’s more clear to me now. I was under the impression people would need to manually run multiple things. I think so long as we have proper documentation and guides it won’t really be an issue, but we shouldn’t try and obfuscate things with automated scripts/packages.
1 Like
I was thinking it would just copy the binaries into some standard $PATH location (or can be an argument passed into the script). This should work on all POSIX systems. For secure mode we will only support Linux x86-64 – if we detect that, only then would we run all the security-specific stuff we will have in the future.
The script could set some bit on the system (e.g. a file at ~/config/polkadot) to signal that it’s run. Once we implement secure-mode, validators would start getting an error on startup if they haven’t run the script.
Workflows like zombienet testing could also benefit from this. When I make a change I need to test, right now I need to recompile everything (remembering to use the production profile) and copy the binaries to my $PATH and it’s actually an error-prone process – having a single script to run would help in this scenario as well.
We could perhaps provide such a script in addition to a guide to automate things for people. But this is all just the vague idea I had, no strong opinions here either way. Just keep in mind that we will be doing security stuff in the future, including maybe configuration of cgroups, which is notoriously difficult to understand and hard to use. Maybe it would be more secure and less prone to error if we automated things for validators, at least the secure-mode stuff?
Got it. But as a number of people (like me) develop and test on Macs, we need basic support for it, and ideally functionality shouldn’t diverge significantly from Linux. We have tried to keep the single binary UX by embedding the other binaries into polkadot and spawning processes from memory, but it ended up working significantly differently on Mac, which makes it harder to develop and catch issues for Linux.
AppImage and Flatpak look interesting. Do they allow running arbitrary commands as part of installation? I’m agnostic to what specific strategy we pick in the end, but I do wonder what others think about this approach.
That is also no “native” approach. That is basically the same as a docker container, just in a different form.
The entire discussion around what to use is really not bringing us anywhere. You could also just talk about if you should create an arch package, an ubuntu package or a centos package. If there is a guide, you can go out and adapt and write all the packages that you like. Or just let it be done by someone else 
I recommend this. An installation script that works for most users and a guide because it will very likely break for some. I say this also as a nixos user, who’s not smart enough to make things work on my own a lot of the time 
We should also have a good story for testing:
- Plain running of nodes locally.
- Pain free integration with Zombienet.
- Deployment on Versi
1 should be fairly simple, as we will have standardized locations within the target directory. We should just make sure to have a simple cargo target that builds anything relevant. Same should apply for Zombienet. For Versi, the workflow will need adaption. Right now we just point it to the Polkadot binary - this will no longer work.
Right now I am just thinking the installer can be a thin wrapper around cargo install. That way users can still pass it env vars for configuration and we can always expand our installer script or swap out cargo install if needed. This keeps things simple and future-proof. A bit more detail here.
Does it meet the requirements you mentioned? (I am not sure how installation works with Versi right now.) It would be great if someone from the release team also weighed in on requirements.
1 Like
Ideally we would not need a script at all for local testing: only relying on cargo features. Does that work? If not what are blockers?
Please write a guide, no script! It is like I’m talking against a wall.
This would already solve this issue and you would not need to care at all…
Do you mean for zombienet and such? With cargo you can do cargo install --path ..
I raised Gather requirements for binary distribution · Issue #7284 · paritytech/polkadot · GitHub. I want to gather all our hard requirements in one place. cc @will as you would know best for the validator side. I’ll get an answer for Versi.
These are mainly used for desktop apps (snap and Canonical are notable exception) and the main benefit is that you can ship specific dependencies to your app so that you avoid making builds for N different distros with N different environments. Optionally they provide some sandboxing which you either use or ignore. Appimage is self executable but flatpak requires additional environment.
I think they are both overkill for distributing multiple binaries and putting them on the right place.
1 Like
Here’s my proposal for the guide:
To install from the repo:
cargo install --path .
To install straight from GitHub:cargo install --git https://github.com/paritytech/polkadot
To change the installation directory, pass the--rootoption.
FYI: In the future we may need a separate section or separate guide for secure validators, since they’d have to build with musl, set up some security config, etc. But for now we can keep the scope small.
Where should the different binaries being stored? How can someone overwrite the position on where to find the worker binaries. What about people who download the binaries from the release?
Where should the different binaries being stored?
Right now the official docs don’t mention where they should be stored, AFAICT.
How can someone overwrite the position on where to find the worker binaries.
Did I answer this with the --root option or do you mean something else? If you mean overwriting binaries that are already there, should we include the --force option in the guide?
What about people who download the binaries from the release?
Good question. I wasn’t sure if this should be supported anymore. With more binaries it’s harder and more error-prone. Maybe we can keep it in the guide for people who don’t want to (or can’t) build themselves, but make this option less prominent.
My suggestion would be to change the order with “building from sources” first as it’s the easiest, then “installation with package managers”, and finally “installation from official releases”. At the end there should be a new “validate the installation” section. Feedback welcome.
No, the node binary will need to find the worker binaries and the assumption should not just be, they are in the same folder as the node binary.
Which is still the reason I would support packing all binaries into the same binary 
1 Like
Oh, I was under the assumption that validators would put the binaries in $PATH. I can make this explicit in the guide. Otherwise they would need to pass e.g. a --worker-path flag to polkadot. (Which we should have anyway.)
I’ll raise a draft PR later that people can comment on.
Which is still the reason I would support packing all binaries into the same binary
Well I still would support an actual script. But let’s not re-tread the same ground, we reached a good compromise I think.
You still don’t get that I never said that you can not write your script. You can write as many scripts as you like. I don’t care. The point being, that there should be a guide for people so they are not required to read these shitty bash scripts that are full of hacks. And this guide have the priority, what you do afterwards is up to you.
I raised a draft PR with the proposed guide changes:
Note that I changed something from my proposal in this thread. Earlier I wrote,
My suggestion would be to change the order with “building from sources” first as it’s the easiest, then “installation with package managers”, and finally “installation from official releases”.
But, after reading the existing guide, I saw that there are extra steps that make building from source harder. So, I kept the order as-is, and marked “building from sources” as optional. I also wrote,
At the end there should be a new “validate the installation” section.
and I did make this change, so that validators ensure all binaries are in the $PATH.
@will Can I please ask you to take a look? And, can you suggest some others who should review the proposed changes?