Pop CLI Developer Survey · May 2025

1

Screenshot 2025-05-30 at 16.57.19
Screenshot 2025-05-30 at 16.57.191412×800 50.2 KB

We ran a developer survey to better understand how teams are using Pop CLI, where they’re getting stuck, and what features they need most to build in the Polkadot ecosystem.

Some standout findings:

  • 50% of developers requested more smart contract templates, especially for multichain use cases using XCM or ISMP.
  • Cross-chain testing was the #1 frustration across workflows, and fork-based simulation tools were the most requested testing feature (60%).
  • Developers are seeking audited templates, improved docs & examples, full-stack frontends.
  • 80% believe Pop CLI should be funded as a common-good tool to support the broader ecosystem.

For key takeaways go to the end.

1. “Which best describes your current role? (Select all that apply)”

  • Parachain (rollup) / Polkadot SDK Developer – 21
  • Full-stack dApp Developer – 10
  • Frontend Developer – 8
  • Product Manager / Founder – 7
  • ink! Smart-Contract Developer – 7
  • Tooling / Infrastructure Developer – 5
  • Just exploring Polkadot development – 5
  • DevRel / Ecosystem Engineer – 4
  • Developer from another ecosystem (e.g., Ethereum, Cosmos, Solana) – 2
  • Security Researcher – 1

A majority self-identify as parachain and SDK builders. Pure ink! contract-only or exploratory devs are under-represented, hinting that outreach outside polkadot chain dev circles could improve sample diversity.

2 a. “How familiar are you with the Pop CLI?”

  • I’ve used it occasionally (a few times) – 20
  • I’ve heard of it, but never tried it – 14
  • I use it regularly (daily/weekly) – 5
  • I haven’t heard of Pop CLI before this survey – 1

Half the respondents have tried Pop CLI at least once, but only 12 % use it weekly. Awareness is healthy; depth of adoption is not.

2 b. “What best describes why?” — 40 total responses

  • Haven’t had time to try it – 16
  • Blank – 13
  • Unsure what it’s for / need more info – 6
  • Using other tools or current tasks don’t require it – 3
  • I use it only for a specific purpose – 1
  • Other unique comments – 1

Lack of time and clarity of purpose outweigh technical blockers. Better docs, demos, and quick-wins could nudge the “lurkers” into actual users.

3. “Which features have made the biggest positive impact on your chain-development workflow? (Select all that apply)”

  • “I haven’t used the CLI for chain development” – 19
  • pop new parachain (start a new chain project) – 14
  • pop up network (launch a local testnet) – 14
  • pop build (build/check/optimize a runtime) – 8
  • pop call chain (interact with chain functions) – 8
  • pop test (test runtime upgrades) – 5
  • Wallet signing for deployment/interaction – 3
  • pop deploy (deploy to a relay chain) – 2
  • pop bench (benchmarking) – 1
  • Prefer native SDK tools instead – 1

Offering up to date and working templates, as well as easily bootstrapping local networks are the most valued.

4. “What specific tasks or stages slow you down or frustrate you the most in chain development?”

Cross-chain testing — 9 mentions

  • “Cross-chain testing”
  • “I think cross-chain testing”
  • “Cross-chain testing, runtime upgrades”
  • “cross-chain testing & benchmarking”
  • “cross-chain testing”
  • “Cross-chainn testing” (typo in original)
  • “Benchmarking and cross-chain testing”
  • “deployment, cross-chain testing”
  • “cross-chain testing, benchmaring”

Benchmarking — 6 mentions

  • “Benchmarking for sure.”
  • “benchmarking”
  • “Get scheduled actions to work in benchmarking tests, especially when using hooks like on_idle”
  • “cross-chain testing & benchmarking”
  • “Benchmarking and cross-chain testing”
  • “benchmarking, spinning up testnets”

Runtime upgrades — 4 mentions

  • “runtime upgrades”
  • “Cross-chain testing, runtime upgrades”
  • “Runtime upgrades, dependencies, breaking changes, lack of documentation”
  • “SDK upgrade, e2e testing, security hardening”

Integration / end-to-end testing — 5 mentions

  • “E2E testing, for example with Chopsticks. It just takes so long to set it up.”
  • “Polkavm integration”
  • “Integration testing… Some things cannot be tested just with unit tests…”
  • “initial setup, pallets integration (a lot of issues because of wrong versions)”
  • “SDK upgrade, e2e testing, security hardening”

Setup / tooling complexity or poor docs — 4 mentions

  • “the documentation :’)”
  • “Runtime upgrades, dependencies, breaking changes, lack of documentation”
  • “Integration testing… powerful what you can do in the node.”
  • “initial setup, pallets integration… nothing to do with real tasks.”

Deployment — 2 mentions

  • “deployment, cross-chain testing”

Other unique frustrations — 5 mentions

  • “Not all new Polkadot-SDK features have corresponding RFC (e.g., claim queue)”
  • “Start a solo/parachain locally”
  • “local testing”
  • “The complexity of the framework. The lack of a clear dev journey”
  • “debugging in Substrate is still an unease experience”

Cross-chain testing is the single biggest drag, followed by benchmarking and runtime upgrades. These are mostly ecosystem-level gaps more than pure CLI gaps.

5. “Which CLI features would be most valuable to your chain-build workflow? (Select all that apply)”

  • Omninode support (shared runtimes, hybrid use cases) – 16
  • Audited chain templates (maintained by R0GUE) – 15
  • Frontend templates for full-stack dApps on top of your chain – 15
  • Auto-upgrade support (psvm integration, transaction-version bump detection) – 13
  • Other utilities (address generation, key management, signing messages) – 6

Demand clusters around templates (omninode + front-end) and safety rails (audits, auto-upgrades).

6. “Which CLI features would be most helpful during testing and iteration? (Select all that apply)”

  • Dry-run extrinsics, fork chains, simulate live state (e.g., Chopsticks or Zombie-bite integrations) – 24
  • View on-chain state (inspect storage keys via pop call chain) – 20
  • Chain fuzzing ( pop test with randomised inputs to uncover runtime bugs) – 15
  • Fee Reporter (estimate weight → fee) – 11

Simulation & state-inspection are clear front-runners; they dovetail with the cross-chain-testing pain noted earlier.

7. “Which CLI features would improve your deployment experience? (Select all that apply)”

  • Enhanced PDP integration (auto-generate API keys, runtime-upgrade & lifecycle ops helpers) – 17
  • Interactive Network Config Generator (.toml wizard…) – 14
  • Integration with external infra providers (e.g. Zeeve’s Perfuse) – 7
  • Production-ready deployment flow (reference Parity Helm Charts) – 1
  • Custom allocator support for Polkavm contracts – 1
  • Placeholder / “x” responses – 13

Ops automation (PDP, config wizards) outweighs infra-provider integrations. Many skipped this question, possibly reflecting lower maturity in deploying to prod.

8. “Which features have made the biggest positive impact on your contract-development workflow?”

  • I haven’t used Pop CLI for contract development – 18
  • pop new contract (start a new contract) – 16
  • pop up contract (deploy) – 13
  • pop call contract (interact) – 9
  • pop build (build/optimize) – 8
  • pop test (includes E2E testing) – 3
  • Wallet signing for contract deployment / interaction – 1 †

Nearly half of respondents still haven’t touched Pop for contracts. Among users, the create-deploy-interact trio delivers the most value.

9. “Which new contract features are most important for your build workflow? (Select all that apply)”

  • Expand the number of templates (e.g. Multichain) – 20
  • Frontend templates for full-stack dApps… – 15
  • I’m not familiar with these features – 11
  • Other utilities (address generation, key management, signing messages) – 4
  • PolkaVM contracts – 1

Again, templates top the list, reinforcing a theme: builders want curated starting points.

10. “Which new contract features are most important for your test workflow? (Select all that apply)”

  • Simulate transactions and fork a network (Chopsticks integration) — 21
  • View contract state (inspect via pop call contract) — 15
  • I’m not familiar with these features — 10
  • Contract fuzzing (pop test –fuzz) — 6

The appetite for fork-based simulation mirrors the chain side—devs want realistic testbeds without spinning up full networks.

11. “Which new contract features are most important for your deploy workflow? (Select all that apply)”

  • I’m not familiar with these features — 15
  • Verifiable builds … and contract verification — 13
  • Interactive Network Config Generator — 10

Knowledge gap is large (15 “not familiar”); educational content may unlock demand for verifiable builds.

12. “What specific stages or tasks in your smart-contract development are slow, frustrating, or unclear?”

Debugging & error messages — 3

  • “Debugging when the error occurred.”
  • “In Rust the whole IDE turns red…”
  • “Still debugging. The error returned… hard to understand.”

End-to-end testing & live deployment — 2

  • “E2E testing”
  • “Live testing and production deployment”
  • “Smart contract deployment ink6”

Cross-environment integration — 1

  • “EVM intermingling with Substrate and observability”

Error diagnostics stand out—clearer stack traces and messages could pay big UX dividends.

13. “What kinds of ink! contracts are you building (or plan to build)?”

  • Experimental / learning – 19
  • Multichain solutions (XCM or ISMP) – 14
  • DeFi / AMMs – 11
  • On-chain games – 9
  • DAO tooling – 8
  • NFTs – 3
  • Other – Events, Subscriptions, Payout

Exploratory and multichain use-cases dominate, aligning with earlier calls for better cross-chain tooling.

14. “How would you rate the onboarding experience of the Pop CLI?”

  • 1 – Very difficult / Confusing: 0
  • 2 – Somewhat difficult: 2
  • 3 – Neutral / okay: 6
  • 4 – Smooth: 15
  • 5 – Extremely smooth and intuitive: 7

73 % rate onboarding ≥ 4—a solid first-impression score.

15. “What was confusing or missing when you first tried Pop CLI?”

  • “In the docs… had to manually download the chainspec”
  • “Documentation”
  • “I personally haven’t tried using it yet”
  • “Fuzzing”
  • “I could not check the debug string…”
  • “The actual purpose of the tooling”
  • “Show-cases (not just doc)”

Docs & discoverability once again. Quick fixes: pre-baked examples, clearer purpose statement, and providing how-to’s.

16. “How would you rate the user experience of the Pop CLI?”

  • 1 – Very poor: 0
  • 2 – Needs improvement: 1
  • 3 – Average / acceptable: 6
  • 4 – Good: 11
  • 5 – Excellent: 9

Overall UX is good (74 % rate ≥ 4), but note the mild drop from onboarding to day-two usage.

17. “What would make the CLI a 5-star tool for you?”

  • Frontend
  • Chopsticks / Zombie-byte integration
  • More documentation & video tutorials
  • Haven’t tried yet
  • Complete coverage for chain interaction & contract deployment
  • mandoc support (view docs with man)
  • Show-cases + let me test something faster
  • Did not use it enough to give a proper answer

Requests cluster around docs, examples, and deeper integrations rather than brand-new features.

18. “Would you consider the Pop CLI a common-good tool that should be maintained and further improved through the Polkadot treasury?”

  • Yes: 32
  • No: 1
  • Not sure: 7

Strong mandate (80 % “Yes”) for treasury-backed maintenance and evolution.

19. “Do you have any feedback, feature ideas, or integration suggestions you’d like to share with the Pop team?”

  • A front-end visual builder
  • Fuzzing & symbolic-execution integration; multi-contract tests
  • Focus more on PolkaVM
  • Hub of showcases for devs/startups (wallets, indexers, DEX on AssetHub Plaza, etc.)

The community is hungry for visual tooling and a curated library of real-world showcases.

20. “Would you be interested in using a visual builder (Pop Builder) to create smart contracts or chains without writing code?”

  • Yes — valuable for my team: 3
  • Yes — should exist for Polkadot dev: 11
  • Maybe — depends on features: 10
  • No — I prefer CLI tools: 5
  • No — waste of time: 1
  • Not sure: 5
  • Blank: 5

Interest skews positive/curious (24 of 40). A well-scoped MVP could validate demand.

21. “How could Pop CLI better support developers building on or integrating with the Polkadot Hub?”

  • Unsure how ink! will interact with governance, staking, assets…
  • Smooth onboarding flow that covers everything (even faucets)
  • Define clear founder/dev/builder journeys
  • Better PolkaVM support, more macros
  • Compatibility with Solidity contracts
  • Showcases

A consistent theme: end-to-end journeys and cross-toolchain examples are more valuable than individual CLI flags.

What we take forward

  • Prioritise learning materials – Clear, concise documentation, step-by-step tutorials, and video content will do more for daily retention than any single new command. Every audited chain or contract template should ship with its own written guide and companion video.

  • Templates as first-class features – Expand the catalogue (chains, contracts, front-ends) and audit them. Pairing each template with dedicated docs and media turns it into an on-ramp rather than a code sample. In addition, add omninode support to simplify chain templates, development and maintenance.

  • Address the multichain gap – Interest in XCM/ISMP solutions is high, but confidence is low.

    Pop CLI: add state-forking and local-network spin-up for rapid iteration. Also highly requested features

    Outside Pop CLI: writing XCM tests is extremely difficult. The XCM emulator is the best we have. Other options are writing integration tests using chopsticks or zombie-bite. Communication and collaboration with the ecosystem will be initiated to align and improve before mentioned test frameworks (@francisco.aguirre & @acatangiu).

    The ink! Alliance will invest time into improving drink!, a framework to write tests for ink! smart contracts that interact with the runtime. This can potentially be expanded to write cross chain tests.

  • Use telemetry for continuous improvement – Combine CLI usage stats with website analytics to spot drop-offs, then refine docs and templates where they matter most.

  • Lean on community mandate – With verified support to treat Pop CLI as a common good, we will pursue a new treasury proposal—provided we deliver measurable progress.

  • Embed in Polkadot-Hub research loops – work together with Parity on the hackathon surveys, adding focused ink! and CLI questions. Frequent, lightweight feedback cycles will keep the roadmap aligned with what Rust-first smart-contract builders actually need.

8 Likes