Mock-builder: a utility to create mock pallets

Tech Talk
1

I want to present a crate we’ve developed as part of our work in the centrifuge parachain that can be useful for other Substrate projects. We try to give a solution to clear all boilerplate, dependencies, and knowledge needed to set up pallet tests.

Problem

As part of our development, we have needed to create pallets that depend on many other pallets. We always try to follow a loose coupling pattern, but all benefits of this are lost in the testing part. Once you create the mock.rs file, you need to configure your pallet with other pallets, which need to be configured with their dependencies, and so on.

We could say that we are tightly coupled in testing, and most of the effort to make trait-based abstractions are lost. This tight coupling comes with some downsides to the testing part:

  • You need to learn how to configure other pallets used as dependencies.
  • You need to know how those pallets work because they directly affect the behavior of the pallet you are testing.
  • The way they work can give you non-complete tests. It means that some paths of your pallet can not be tested because some dependency works in a specific way.
  • You need a lot of effort maintaining your tests because each time one dependency changes, it can easily break them.

Solution

To solve this, we bring a utility to create mock pallets. We define a mock pallet as a pallet that implements any number of traits whose behaviors can be specified in each test case using rust closures. There are other crates as mockall that mock traits. Instead, this crate gives you an entire functional pallet ready to use in any runtime, implementing any combination of traits.

For example, if we want to build a mock pallet that implements a trait that has a method fn foo(a: u32), this utility implements foo() and builds a new method called mock_foo() that can be used from the test cases as follows:

MyMockPallet::mock_foo(|a| {
    // Mocking your behavior here
});

Each time the pallet calls internally to foo() it will execute the closure we have defined above.

This solves the issues mentioned above, getting loose coupled tests:

  • You no longer need to configure dependencies for testing your pallet.
  • The behavior is defined for each of your use cases.
  • You can call mock_foo() any number of times, returning all possible values to get complete tests.
  • You no longer need to fix your tests if some dependency changes because you have no dependencies!

Test it!

We have used this way of doing testing in our project with successful results, and we want to share the utility to help the community and get feedback to improve it.

Currently, it supports traits with complex generic methods and lifetimes, and we want to focus on improving the mock pallet creation part using procedural macros and removing the automatic boilerplate, which is now needed.

  • You can check the docs, to get a deeper understanding of how it works and how to use it.
  • You can use it in your project as follows (although we will possibly create a new repo to ease the maintainability):
    mock-builder = { git = "https://github.com/centrifuge/centrifuge-chain", branch = "main" }

Happy to hear your thoughts on this!

13 Likes
2

This looks great! Looking forward to trying it out next time I test a pallet.

1 Like
3

This doesn’t always need to be like that though.

Let’s take a simple example: most pallets have type Currency: .... And most of these pallets bring in the full might of pallet-balances in the mock.rs in order to finally express type Currency = pallet_balances::Pallet<Runtime>.

But this is by no means mandatory. You could implement mocks for such interfaces and use them as well. For example, see pallet-nomination-pool, which has type Staking: .. which is supposed to be pallet-staking. In the mock, we provide a fake implementation of this, but in a real runtime, we couple it with pallet-staking

Having read your solution, I believe what you have proposed is a framework to simplify the above process, namely in the case where you want your “custom mock pallet impl” to have more complex logic?

Perhaps it would also be useful to demonstrate how mock_pallet can be used for the above two examples? And we could even take this to a Substrate PR and use it eg pallet-nomination-pools if suitable :slight_smile:

4 Likes
4

Sure! Using the real implementation is not mandatory. You can always implement your own mocks for testing your pallet. Nevertheless, it’s quite difficult to choose what to return or expect based on different use cases because these implementations are static and fixed to one behavior. When you need to implement different versions of the mock for different test cases, this becomes very tedious, and in a lot of cases, you end up using a real implementation that is “more” configurable/flexible.

It’s true that using storage in parameter_types make this process simpler, but it still need to be implemented manually for each trait.

As you have said, you can see this utility as a framework to do this tedious job for you, and also provide more flexibility. Since the future intention is to generate these mock pallets automatically, it can also be worth for mocking simple logic.

Good idea! I’ll open a PR using mock-builder with this pallet in order to show how it works in a real scenario :rocket:

5

The trait impls used in testing can be as sophisticated as needed. See for example, the xcm-builder mock and the auctions mock.

It would be interesting to know how much the mock-builder simplifies this pattern.

3 Likes
6

Any comparison of this with some generalized Rust mocking tool?

7

Yes! mockall, for example, widely used in rust, mocks traits. It means you declare a type, and mockall implements the traits you want for that type. Nevertheless, to get working traits with static methods, it needs to use static memory to place what they call “expectations” (the closures you want to run once the method is called), so you can not use the mock twice in different pallets, or testing in parallel without having concurrent issues.

Instead, mock-builder mocks pallets. So it implements traits for the pallet itself (the Pallet<T> type). This gives you all the power that comes with pallets, such as having several instances or running the tests in parallel. Also, the user does not need to handle the lifetime of “expectations” because it is done automatically by the pallet’s lifecycle, so the API can be much more straightforward: just defining a closure with what you expect.

2 Likes
8

As a draft, here is a PR showing how it would look in real pallet testing. From the links above, I’ve chosen the auctions pallet.

The main difference between the TestLeaser and MockLeaser is that TestLeaser is not a mock itself. It’s a simpler implementation used for tests. It doesn’t allow you to specify the whole behavior of the method called by the pallet. This could lead to not getting full coverage (because the implementation does what it does), or it could lead to creating a false feeling that you’re testing real code when instead, you’re testing your own TestLeaser implementation.

More than simplifying things, the big point of using a mock pallet is that you can redesign how the test cases are done, focusing on checking PRE/POST conditions of methods and getting better coverage and less overlapping testing behavior.

Of course, you could extend TestLeaser with more thread_local storages to improve it and make it more flexible, handling each input/output method parameter. But this comes with a lot of effort and boilerplate for the programmer; that is just what mock-builder does for you (automatically, once we add the procedural macros).

To summarize what I think are improvements in this pattern, comparing with the use of custom trait implementations for tests:

  • The mock pallet can be built automatically (by procedural macros), so you do not need to think about how to implement i.e. TestLeaser.
  • You can reach better coverage. You can specify what to return in each use case to ensure every path is tested.
  • You test the real code. You won’t get confused about testing the testing code.
  • You can organize tests without overlapping testing behavior. Tests can be more isolated, focusing on testing only one part of the logic without initializing previously required states.
  • Data in thread_local can live more than the test case, so it’s easy to write from one test and read a wrong state from another if running in the same thread, making the process of debugging tests difficult.
  • Data stored in thread_local can not be generic, so you can not reuse TestLeaser implementation for testing other pallets that use different types. Instead, by using pallets, you can configure them through the Config trait allowing you to reuse the generated mock pallet across several pallet tests.
2 Likes
9

PR example reopened here: Mock builder: usage example for the auction's Leaser type by lemunozm · Pull Request #7406 · paritytech/polkadot · GitHub

10

It has been a few months since our last interaction in this space. Given the positive response we received, we are excited to invite you to provide another round of feedback. We have successfully completed V1 of the mock-builder tool [1], which is now available outside the Centrifuge organization, residing under the newly established Free Open Source Software for Web3 (foss3) organization [2].

We are aware of the fact that the true potential of the mock-builder will be realized in V2, with the inclusion of procedural macros. These macros will effectively reduce the burdensome boilerplate code and introduce valuable automations, greatly enhancing the user experience. However, it’s important to note that implementing such a feature cannot be accomplished within a short timeframe, such as a single week. Therefore, we earnestly request community feedback to help us explore our next steps. If anyone would like to see second application of mock-builder against another pallet, please let us know!

We firmly believe that V2 will provide significant benefits to all Substrate builders, simplifying the process of conducting precise isolated unit testing and ultimately leading to improved test coverage. Your input highly appreciated!

Directly pinging you @bkchr since the conversation in the exemplary draft PR in the Polkadot repo was forced to an hault due to the monorepo migration. Unfortunately, it’s not possible to keep conversation history when migrating PRs.

1 Like
11

Is there a substantive reason this wasn’t added under:

12

New updated docs can be found here: mock_builder - Rust

We’ll soon provide a more real-like pallet example to show the use case better

13

That’s a fantastic question you’ve brought up! We’re totally on board with the idea of incorporating this into ORML. In fact, your comment has inspired me to kick things off by opening an RFC [1] in their repository.

We went ahead and set up our own repository, and we’re absolutely welcoming external contributors with open arms. The reason behind this move is that we’ve got some other pallets in the pipeline, ready to make the leap from the Centrifuge repository to our new home. Our hope is that these pallets will eventually become just as popular and beneficial as the ones already living in ORML.

The main reason we took this route is that, without write access or an existing CODEOWNERS file in ORML, we didn’t want to dump all the reviewing and maintenance work on the shoulders of the ORML maintainers. That wouldn’t have been fair or cool at all. So, setting up our own spot seemed like the way to go. Thanks for understanding!

14

Thanks, I see they have responded positively to your approach.

15

A complete pallet example showing how to to use mock-builder in a real scenario can be found here

1 Like