TransactionExtension, the replacement of SignedExtension, has finally landed on polkadot-sdk/master.
The introduction of TransactionExtension, the first step of the Extrinsic Horizon long term vision, unlocks the true potential of Polkadot transactions while addressing many shortcomings of the previous SignedExtension model.
In the previousSignedExtension model, the runtime defined a SignedExtension, usually a pipeline of standalone SignedExtensions wrapped as a tuple. The purpose of these extensions was to add data to transactions, both implicit (e.g. the genesis hash of the chain in frame_system::CheckGenesis) and explicit (e.g. the nonce in frame_system::CheckNonce), and to provide a framework of validating transactions (e.g. disallow transactions from the 0 address in frame_system::CheckNonZeroSender).
Now, let’s go through some of the main changes coming with TransactionExtension.
Weights on transactions
Previously, there was a convention around SignedExtensions to keep them light in terms of computation with little to no branching paths in their logic. This was because there was no way to properly measure and charge the appropriate amount of weight to account for each extension’s cost. To prevent spam, an ExtrinsicBaseWeight was added to each extrinsic, a weight which was benchmarked as the average weight of a signed remark extrinsic for the runtime.
TransactionExtension introduces weights to extensions: worst case weights estimated before dispatching anything as well as weight refunds post-dispatch - all built on top of the current weight infrastructure and natively supported in frame_support and the construct_runtime macros.
Origin authorization
The model we’re leaving behind
Previously, there were two types of extrinsics:
- Unsigned extrinsics:
- Can be either inherents or unsigned transactions.
- Are typically dispatched with the
frame_system::RawOrigin::Noneorigin. - Go through the
ValidateUnsignedimplementation of the runtime. - Also go through the extension pipeline, but without any implicit or explicit data, so a much more restricted validation scope.
- Signed transactions:
- Are dispatched with the
frame_system::RawOrigin::Signed(_)origin. - Use an account to sign the transaction.
- Have the signature verified as the first step in ensuring transaction validity.
- Go through the transaction pipeline, which would typically have extensions to ensure the signer has funds to pay for the fees (at least initially, until the refund was awarded) and provides a nonce to be checked and incremented.
- Are dispatched with the
Introducing General transactions
General transactions were introduced in RFC84 to provide a way to support transactions which obey the runtime’s extensions and have according extension data yet do not have hardcoded signatures. This shift in paradigm inherently means that, in order for these transactions to be safe for the chain, the authorization responsibility is moved from the hardcoded signature verification found in traditionally signed transactions to any of the extensions in the pipeline.
What the future looks like
Powerful authorization
The key feature of the new TransactionExtension interface is bespoke transaction authorization, beyond the current limited scope of signed and unsigned transactions. Extensions can now mutate the origin during transaction validation and so can authorize any origin defined by the pallets configured in the runtime. The validation logic happens in order of each TransactionExtension’s position in the extension pipeline, and each subsequent call to validate receives the origin mutated by the previous extension.
The only constraint for a valid transaction is that it must have an authorized origin after the extension validation is finished and the call is to be dispatched; in other words, some extension must lend credibility to this transaction by mutating its origin. In the new paradigm, the signature in signed transactions can be thought of conceptually as the first TransactionExtension hardcoded in the extension pipeline which gives it the frame_system::RawOrigin::Signed(_) origin.
This will allow any substrate chain to create extensions which authorize an origin through custom signature schemes, using different signing algorithms than the ones defined by the runtime, over any on-chain state. This origin authorization, used in tandem with the new General transactions, will allow for truly free transactions in the context defined and authorized by the runtime, where no pre-existing or funded account would need to exist in order to run some logic on chain, provided some extension authorizes it. In a General transaction, if no extension authorizes an origin, the transaction is not valid and will not be included in the transaction pool or dispatched, as the validation will fail.
Heavy lifting in extensions
The introduction of weights to the extensions now allows even authorization mechanisms which are compute heavy, like ZK proofs, to run in the extension context. Previously, extensions were weightless and, by convention, they had to perform minimal compute in order to limit this attack vector.
Additionally, the initial assumed weight of an extension is not static, but rather computed using both the extension state as well as the call that is to be dispatched. This allows for a finer tuning of the transaction’s initial weight for extensions which could be really expensive in some cases but cheap in others.
One such case is ChargeAssetTxPayment, which could have 3 substantially different weights, in descending order:
- payment with an asset through asset conversion
- native payment
- no payment - the previously assumed weight will be refunded in
post_dispatch
This allows the runtime to register a weight amount close to the final value, rather than every transaction incurring the cost of the greatest weight and having to wait until post_dispatch for a refund, potentially missing out on block inclusion.
Migration of current extensions
The existing extensions in polkadot-sdk have been migrated to the new interface and now have accompanying benchmarking code, where applicable.
Going forward, all extensions must handle origins different than the traditionally signed origin and establish, on a case-by-case basis, which origins should trigger validation and/or state mutations for a particular transaction.
Among all migrations, two important extensions, which protect the chain from spam, stand out:
CheckNoncenow checks and increments the nonce only forframe_system::RawOrigin::Signed(account)origins.ChargeTransactionPayment/ChargeAssetTxPaymentnow require enough funds to pay for the transaction and actually charge that amount only forframe_system::RawOrigin::Signed(account)origins.
These changes basically suggest that the payment, both for the nonce (through funding an account with at least the existential deposit) as well as the transaction fee, is required only for account origins, which must come from traditionally signed transactions, so they must have a frame_system::RawOrigin::Signed(account) origin. A nonce is stored in the frame_system::Account map under an AccoundId, and payment in any asset can only be extracted from an account, as the funds must belong to someone who is willing to give them up (proven through a signature, i.e. an account which signs a transaction).
Unsigned transactions, with origins
The current unsigned extrinsics refer to both inherents and unsigned transactions.
Inherents are extrinsics created by the runtime itself which the block author must include in their block, at the very beginning. The protocol mandates this, and they must be included regardless of who is the block author, so they have no origin and nobody is charged for their inclusion.
As of today, unsigned transactions are transactions validated by the runtime through ValidateUnsigned, but which have no particular origin. Conceptually, this works well for calls such as apply_authorized_upgrade, where, once a runtime upgrade was authorized by storing the future code’s hash, anyone should be able to submit the actual runtime code in an apply_authorized_upgrade call and they should not be charged anything, they should not need a funded account. Moreover, the system authorized the upgrade, so the system should enact it. No particular user/account should carry the burden of stamping their name on a particular runtime upgrade.
These transactions today effectively do not have an origin within the runtime, as it is ignored, but they are still valid. That is because while their origin is currently frame_system::RawOrigin::None, they are in fact authorized to run, through the validation in the pallet specific ValidateUnsigned. Essentially, the pallet validates and authorizes calls marked as unsigned.
With the custom origin authorization in TransactionExtension, this process is made consistent by making General transactions for these calls which have a particular extension in the pipeline authorizing them to run on chain. The extension’s validation logic can access all pallet state, mutate it even, decide if/how much to charge as payment or deposit. If the extrinsic is deemed valid, the extension should mutate the origin such that when the call is actually dispatched, it has the expected origin.
More on using this mechanic in phase 2 of Extrinsic Horizon, where ValidateUnsigned will be deprecated and replaced by this new authorization mechanic.
Free transactions
The changes described so far lay the foundation for free transactions. To be specific, a free transaction is understood to be a transaction which, if valid, can be submitted, pass transaction pool validation, be included in a block and execute successfully without involving a funded account, not even an existential deposit.
Today, this could partially be achieved through unsigned transactions, but, as stated above, that comes with massive limitations. For signed transactions, they will always require the submitter to have a nonce stored on chain for the transaction to be valid and to provide the funds for the transaction payment, at least initially, both of which require funding.
A runtime dev can enable this use case using TransactionExtension, which will be demonstrated below on a simple example.
Onboarding users through tickets - free transactions exemplified
Let’s suppose we have a pallet where users can reserve some of their currency in tickets of arbitrary value by registering a public key of arbitrary type. The user that registered this also has the associated secret key. They can then reveal the secret key to someone else, who can in turn claim the ticket along with its deposit by providing a proof which could only be computed using the secret key.
Now that the basic mechanism is established, let’s set some other rules:
- The account that claims the ticket must not exist on chain (to demonstrate the “free” aspect of the transaction). In effect it’s an invite to the chain with some free credit attached, generously provided by the user that registered the ticket.
- Once an account claims a ticket, it can never again claim another one under the same account (to introduce some state mutation into the mix and also not have to deal with preventing transaction replay).
How to create a free transaction - step by step
Let’s go through all the steps to create a free transaction using the new features in TransactionExtension. The steps will be accompanied by code snippets where they will be applied on the example use case described above.
- Create a call that they would like to be free for anyone that meets some conditions.
/// Map of all registered tickets and their value.
#[pallet::storage]
pub type Tickets<T: Config> = StorageMap<
_,
Twox64Concat,
PublicKey,
Balance,
>;
/// Map of all previous ticket claimants.
#[pallet::storage]
pub type Claimants<T: Config> = StorageMap<
_,
Twox64Concat,
T::AccountId,
(),
>;
/// Call to claim a ticket's value into an account that doesn't exist yet and has never
/// claimed a ticket before.
#[pallet::call_index(0)]
pub fn claim_ticket(
origin: OriginFor<T>,
ticket: PublicKey,
destination: T::AccountId,
proof: Proof,
) -> DispatchResult {
let _ = ensure_signed(origin)?;
// Check that the ticket is registered.
let ticket_value = Tickets::<T>::take(&ticket).ok_or(Error::<T>::NoTicket)?;
// Check that the account doesn't exist yet.
ensure!(!System::<T>::account_exists(destination), Error::<T>::AlreadyExists);
// Check that the account hasn't yet claimed a ticket.
ensure!(!Claimants::<T>::contains_key(&destination), Error::<T>::AlreadyClaimed);
// Validate that the proof is correct and the claim is valid.
ensure!(Self::validate_signature(ticket, proof, destination), Error::<T>::BadProof);
// Transfer the funds held in the pot.
T::Currency::transfer_on_hold(
&HoldReason::Ticket.into(),
&T::PotAccount::get(),
&destination,
ticket_value,
Exact,
Free,
Polite,
);
let _ =
T::Currency::release(&HoldReason::Ticket.into(), &destination, ticket_value, Exact);
// Register the claimant
Claimants::<T>::insert(&destination, ());
Ok(())
}
- Create a custom origin within their pallet which carries relevant authorization information.
#[pallet::origin]
#[derive(Clone, PartialEq, Eq, RuntimeDebug, Encode, Decode, MaxEncodedLen, TypeInfo)]
pub enum Origin<T: Config> {
Claimant(T::AccountId, Balance),
}
- Enforce that the origin that dispatched the call matches that custom origin.
#[pallet::config]
pub trait Config: frame_system::Config {
type EnsureClaimant: EnsureOrigin<
Self::RuntimeOrigin,
Success = (Self::AccountId, Balance),
>;
}
// snip
#[pallet::call_index(0)]
pub fn claim_ticket(origin: OriginFor<T>) -> DispatchResult {
let (claimant, ticket_value) = T::EnsureClaimant::ensure_origin(origin)?;
// Transfer the funds held in the pot.
T::Currency::transfer_on_hold(
&HoldReason::Ticket.into(),
&T::PotAccount::get(),
&claimant,
ticket_value,
Exact,
Free,
Polite,
);
let _ =
T::Currency::release(&HoldReason::Ticket.into(), &claimant, ticket_value, Exact);
// Register the claimant
Claimants::<T>::insert(&destination, ());
Ok(())
}
- Create an extension which validates that the conditions for the transaction to be free are met, without mutating the state.
/// An extension which, if activated through providing it with a ticket and a proof, will try to
/// turn a signed origin into an `Origin::Claimant`.
#[derive(Clone, Eq, PartialEq, Encode, Decode, TypeInfo)]
pub struct AuthorizeClaimant<T> {
inner: Option<(PublicKey, Proof)>,
_phantom: PhantomData<T>,
}
impl<T: Config> TransactionExtension<T::RuntimeCall> for AuthorizeClaimant<T> {
const IDENTIFIER: &'static str = "AuthorizeClaimant";
type Implicit = ();
type Val = Option<PublicKey>;
type Pre = ();
fn weight(&self, _: &T::RuntimeCall) -> Weight {
if self.inner.is_some() {
T::WeightInfo::auhtorize_claimant()
} else {
Weight::zero()
}
}
fn validate(
&self,
mut origin: DispatchOriginOf<T::RuntimeCall>,
_call: &T::RuntimeCall,
_info: &DispatchInfoOf<T::RuntimeCall>,
_len: usize,
_self_implicit: Self::Implicit,
_inherited_implication: &impl codec::Encode,
_source: TransactionSource,
) -> ValidateResult<Self::Val, T::RuntimeCall> {
// If the extension is inactive, just move on in the pipeline.
let Some((ticket, proof)) = &self.inner else {
return Ok((ValidTransaction::default(), None, origin));
};
// The extension only works with signed origins which it then authorizes. If the origin
// isn't an account that signed the transaction and the extension is activated, the
// transaction is invalid.
let Some(claimant) = origin.as_system_origin_signer() else {
return Err(InvalidTransaction::BadSigner);
};
// Check that the ticket is registered, but don't remove it yet.
let ticket_value = Tickets::<T>::get(&ticket).ok_or(InvalidTransaction::Custom(100))?;
// Check that the account doesn't exist yet.
ensure!(
!frame_system::Pallet::<T>::account_exists(claimant),
InvalidTransaction::Custom(101)
);
// Check that the account hasn't yet claimed a ticket.
ensure!(!Claimants::<T>::contains_key(&claimant), InvalidTransaction::Custom(102));
// Validate that the proof is correct and the claim is valid.
ensure!(Self::validate_signature(ticket, proof, claimant), InvalidTransaction::BadProof);
// snip
}
}
- If the validation passes, mutate the origin to the custom origin created previously.
impl<T: Config> TransactionExtension<T::RuntimeCall> for AuthorizeClaimant<T> {
// snip
fn validate(
&self,
mut origin: DispatchOriginOf<T::RuntimeCall>,
_call: &T::RuntimeCall,
_info: &DispatchInfoOf<T::RuntimeCall>,
_len: usize,
_self_implicit: Self::Implicit,
_inherited_implication: &impl codec::Encode,
_source: TransactionSource,
) -> ValidateResult<Self::Val, T::RuntimeCall> {
// snip
// Construct an `Origin` as declared above.
let local_origin = Origin::Claimant(claimant, ticket_value);
// Turn it into a local `PalletsOrigin`.
let local_origin = <T as Config>::PalletsOrigin::from(local_origin);
// Then finally into a pallet `RuntimeOrigin`.
let local_origin = <T as Config>::RuntimeOrigin::from(local_origin);
// Which the `set_caller_from` function will convert into the overarching `RuntimeOrigin`
// created by `construct_runtime!`.
origin.set_caller_from(local_origin);
// Make sure to return the new origin. We also pass the ticket in `Val` so that, when it is
// received in `prepare`, we can remove it from storage.
Ok((ValidTransaction::default(), Some(ticket), origin))
}
}
- Perform any necessary state mutation.
impl<T: Config> TransactionExtension<T::RuntimeCall> for AuthorizeClaimant<T> {
// snip
fn prepare(
self,
val: Self::Val,
_origin: &<T::RuntimeCall as Dispatchable>::RuntimeOrigin,
_call: &T::RuntimeCall,
_info: &DispatchInfoOf<T::RuntimeCall>,
_len: usize,
) -> Result<Self::Pre, TransactionValidityError> {
// If the extension was active during validation, remove the ticket that will be claimed.
if let Some(ticket) = val {
let _ = Tickets::<T>::take(&ticket);
}
Ok(())
}
}
Other resources
For an example on how to use the new authorization feature through custom signing schemes, origin mutation, integration in a TransactionExtension pipeline and more, check out the authorization-tx-extension example in polkadot-sdk.
What’s next
Right now we’re working on the finishing touches of the TransactionExtension effort and making sure everything is available in the upcoming December 2024 polkadot-sdk stable release.
The introduction of TransactionExtension is only the first step of Extrinsic Horizon, with phase 2 already in the works. Phase 2 will deprecate ValidateUnsigned and migrate all unsigned transactions to the new General transaction format through a framework for authorization built on TransactionExtension. Once delivered, this will separate inherents from all other transactions and also provide a handy way to do transaction authorization without signatures. For more details, check out the Extrinsic Horizon issue as well as the draft PRs on this topic.
) are unchanged and here to stay for a while. Moreover, the metadata will continue to expose v4 as the extrinsic format version, with v5 being “experimental”. This means that Ledger and all other mechanisms will continue to work as before for signed transactions.