Skip to content

how-to-implement-new-sdk-components.md

Latest commit

 

History

History
267 lines (186 loc) · 20 KB

how-to-implement-new-sdk-components.md

File metadata and controls

267 lines (186 loc) · 20 KB
Raw

How to Implement New SDK7 Components

This is the high-level steps that must be followed:

  1. Create the protobuf definition in Decentraland's SDK protocol
  2. Generate the new protobuf TypeScript code in the js-sdk-toolchain (serialization test + (optional) sdk helpers)
  3. Generate the new protobuf C# code in unity-explorer + implement the systems that handle the functionalities, cleanup, etc. for the component
  4. (Ideally) Create an example test-scene to show how the component is used from the SDK side

IMPORTANT This is the order in which PRs must be merged:

  1. Merge Protocol repo PR (if the change was on main, then experimental must sync the new changes before step 2)
  2. Update both js-sdk-toolchain repo PR and unity-explorer repo PR to use the updated @dcl/protocol@experimental package
  3. Merge js-sdk-toolchain and unity-explorer in any order

1. Create the protobuf definition in Decentraland's SDK protocol

  1. Create the protobuf definition inside https://github.com/decentraland/protocol/tree/main/proto/decentraland/sdk/components
  2. Create a PR with the new changes

NOTE: After creating the PR, a GitHub Bot will comment with the package link to test the PR. You can use that link for testing in the following steps.

Things to take into account:

  • We are using proto 3, so all the definition of the proto must compile with their syntax.
  • Be VERY mindful of the fields and types you create in the Protobuf Message, due to retro-compatibility standards we can never rename or modify any field of an already released Protobuf message (AKA if you are still working on your PRs then you can keep changing stuff, although you will be forced to change the other 2 repo PRs as well of course), as that will break scenes that are already deployed and create conflicts in the SDK and Explorer... if at some point we really need to "update" an already released component field, we should create a new field and flag the old one as deprecated (example).
  • We have some common types that cannot be recreated.
  • The proto should have the basic definition.
  • You must add the following code to enumerate the component with a UNIQUE ID:
import "common/id.proto";
option (ecs_component_id) = 1100;

NOTE: With the repo cloned locally you can run make list-components-ids and make check-component-id ID=xxx to make sure your component is using a unique id. On Windows you can run this instead: bash -c "grep -rh 'option (common\.ecs_component_id)' proto/ --include='*.proto' | sed -E 's/.*= ([0-9]+);/\1/' | sort -n | nl"

IMPORTANT: New components going directly to main should have a 12xx ID (or cover the previous unused IDs). New components going to experimental should have a 14xx ID. DAO's Regenesis Labs (AKA Protocol Squad) experimental feature components will have a 16xx ID. This was defined to avoid ID conflicts when an experimental feature component is taken to main.

Example of .proto:

syntax = "proto3";

import "common/id.proto";
option (ecs_component_id) = 1020;

message PBAudioSource {
  optional bool playing = 1;
  optional float volume = 2; // default=1.0f
  optional bool loop = 3;
  optional float pitch = 4; // default=1.0f
  string audio_clip_url = 5;
}

2. Generate the new protobuf TypeScript code in the js-sdk-toolchain

Clone the sdk repo: https://github.com/decentraland/js-sdk-toolchain

Run the following command at the root of the js-sdk-toolchain project (use @experimental instead if you are working on an experimental feature):

npm install @dcl/protocol@next

Or the command generated by the GitHub Bot in your @dcl/protocol PR (this must be temporal for testing the PRs).

Then run the following commands at the root of the js-sdk-toolchain project:

make install
make build

And push the generated code to your branch.

For the branch PR to be approved and functional you will also need to:

After that is done, the GitHub Bot comment in the PR will expose a test package so that you install that in your test scene and be able to use the new component from the SDK in the scene (see How to connect to a local scene on setting up a local scene and running it).

NOTE: If development was done and merged to experimental follow these steps to port it to main:

  1. Cherry pick the squash-merged commit without committing by running cherry-pick --no-commit <HASH>
  2. Discard the conflicted test snapshots and playground-assets.api.md (the non-conflicted files will already be staged) with git checkout .
  3. Update the protocol package with the main-based PR in protocol for your changes with npm install .... (PR-generated command)
  4. Rebuild everything with make install then make build and then make update-snapshots
  5. Add the new filechanges and commit to your main-based branch

Last-Write-Wins Component (LWW) and GrowOnly-Value-Set Component (GOVS)

In general, any component will be defined by default to be a LWW, so that every update to it through CRDT will overwrite the whole state of it for whoever reads it later. If not specified or needed, always go for a LWW Component approach.

In some cases, specially for some "result components" populated by the Explorer, instead of a LWW, a less-used kind of CRDT component called GrowOnly-Value-Set is needed. That kind of component always holds a collection of all the previous states of the component.

Deeper and more detailed info on those 2 versions of the CRDT components design can be found at the ADR-117. Here are their main differences:

  1. Last-Write-Wins Component (LWW)
    • The component can be overwritten and the scene will read only its last state.
  2. GrowOnly-Value-Set Component (GOVS)
    • This component holds the history of every write in an array (a SDK helper could be useful to implement in order to let the creators avoid the boilerplate. See videoEvents or assetLoad as reference).

IMPORTANT: In order to implement a grow-only-value result component, you will need to list its name inside the file scripts/protocol-buffer-generation/generateIndex.ts in the GROWN_ONLY_COMPONENTS array.

3. Generate the new protobuf C# code in unity-explorer

To generate the C# code from the protobuf files that come from the protocol, go to the scripts path in the root of the unity-explorer repository. And execute the following commands:

npm install
npm run build-protocol

Prerequisite: Node/npm only. build-protocol runs the protoc-gen-bitwise plugin (for the quantized/bit-packed Pulse network state), a dependency-free Node script bundled in @dcl/protocol — no Python or extra packages required.

To upgrade to the latest version of the @dcl/protocol, we should update using:

npm install @dcl/protocol@experimental
npm run build-protocol

NOTE: unity-explorer needs to always use the @experimental protocol package to support all the experimental features, otherwise the Unity project won't even compile due to missing component files, etc.

To test a PR, we can use a URL generated by the GitHub Bot in the @dcl/protocol PR:

npm install "https://sdk-team-cdn.decentraland.org/@dcl/protocol/branch//dcl-protocol-1.0.0-3143233696.commit-45f1290.tgz"
npm run build-protocol

IMPORTANT: After merging a @dcl/protocol PR, we must make unity-explorer point to the updated @dcl/protocol@experimental package (and NOT the PR test package) before merging the unity-explorer PR.

  • If the protocol update happened at main then the experimental branch must sync those new changes first, and then update the unity-explorer PR with the latest @experimental package.
  • If the protocol update happened at experimental then the unity-explorer PR can just use the updated @experimental package.

Implement the new component functionalities

After the C# code for the protobuf component was generated, one or more systems have to be implemented to handle its functionality, following the Arch ECS architecture.

  • Add the partial class corresponding to the new PB component at IDirtyMarker
  • Register the component at the ComponentsContainer, otherwise no system will recognize the component on the ECS entities.

NOTE: You'll see that some special components are registered as "result components". Those are some components designed mainly to be PUT by the Explorer and only READ by the scene (e.g. input result component, raycast result component).

Last-Write-Wins Component (LWW) vs GrowOnly-Value-Set Component (GOVS)

If the Explorer needs to update the component through CRDT for the scene to be able to read its updated values, for LWW components the PUT command is used, but for GOVS components the APPEND command has to be used. Several "result component" systems can be checked as reference, like the WriteEngineInfoSystem for LWW and the VideoEventsSystem for GOVS.

General implementation guidelines

The following must be implemented:

  • Relevant functionalities for the new component
    • Custom non-sdk struct components (can be empty) are normally used to change the state of the entity.
    • If IsDirty needs to be relied upon, for the case of the scene modifying the component values in runtime and reacting to that: prefer injecting ResetDirtyFlagSystem<PBYourComponent>.InjectToWorld(ref builder) in your Plugin — it clears the flag automatically after all systems in the group have run. Only set IsDirty = false manually inside a query when you need granular control (e.g. a system that handles one sub-case and must leave the flag set for a downstream system).
  • Cleanup of the component
    • SDK component removal
    • Scene unloading (IFinalizeWorldSystem implementation)
  • Pausing of component functionality if the scene is not current or player leaves/re-enters the scene:
    • Usage of ISceneStateProvider.IsCurrent to escape the Update() of the system.
    • (more complex and powerful) Usage of ISceneIsCurrentListener to listen to the moments the user enters/exits the scene.

ColliderLayer mask semantics

Several SDK components carry a collision_mask / collisionMask field of type ColliderLayer (defined in decentraland/sdk/components/common/mesh_collider.proto). The full enum is:

Name Value Meaning
CL_NONE 0 No collisions.
CL_POINTER 1 Pointer-ray collisions (mouse hover, click).
CL_PHYSICS 2 Scene-mesh walls / floors / platforms that affect the player's physics. Does NOT target the character itself — see "Unified main-player qualification rule" below.
CL_PLAYER 4 Any player avatar — main player AND remote avatars.
CL_MAIN_PLAYER 8 The local (main) player avatar only.
CL_RESERVED3..6 16/32/64/128 Reserved — do not use.
CL_CUSTOM1..8 256..32768 Scene-defined custom layers (8 of them).

Additive avatar semantics. The main player capsule is "tagged" with both CL_PLAYER and CL_MAIN_PLAYER: a mask containing either bit matches it. Remote avatars are tagged only with CL_PLAYER. Consequently:

Mask Main player matches? Remote avatar matches?
CL_PLAYER yes yes
CL_MAIN_PLAYER yes no
CL_PLAYER | CL_MAIN_PLAYER yes yes

Unified main-player qualification rule. Both Raycast (ExecuteRaycastSystem.DoesHitColliderQualify) and TriggerArea (TriggerAreaHandlerSystem.PropagateResultComponent) qualify the main player using the same constant:

PhysicsLayers.PLAYER_QUALIFYING_BITS = CL_PLAYER | CL_MAIN_PLAYER

A scene mask that contains either of those two bits qualifies the local player. Any other mask (CL_PHYSICS, CL_POINTER, CL_CUSTOM*, CL_NONE) does NOT qualify the main player on either system.

CL_PHYSICS is deliberately excluded from this set. Per the proto definition, CL_PHYSICS targets scene-mesh walls and floors that affect the player's physics — not the character itself. This matches the convention of every major game engine:

  • Unreal: WorldStatic / WorldDynamic vs the separate Pawn channel.
  • Unity (idiomatic): project layers like Environment vs Player/Character.
  • Godot: world geometry on its own layer; characters on a separate Player layer.
  • Source / Half-Life 2: MASK_NPCWORLDSTATIC (world without entities) vs masks that opt entities in.

Scenes that need to detect the local player on a raycast or trigger area must opt in via CL_PLAYER or CL_MAIN_PLAYER explicitly.

Remote avatars qualify only on CL_PLAYER. They have no client-side physics body either way.

TriggerArea targetOnlyMainPlayer optimisation. When the scene mask is exactly CL_MAIN_PLAYER (no other bits), TriggerAreaHandlerSystem.SetupTriggerArea sets targetOnlyMainPlayer = true. The underlying SDKEntityTriggerArea MonoBehaviour then short-circuits any OnTriggerEnter/OnTriggerExit whose collider transform isn't the main player's, dropping remote-avatar overlaps before they reach the per-frame query in PropagateResultComponent. Any other mask (including CL_PLAYER | CL_MAIN_PLAYER) must NOT enable this optimisation, because it would also reject the colliders the mask is meant to match.

Scene-mesh routing (MeshCollider / GltfContainer). PhysicsLayers.TryGetUnityLayerFromSDKLayer picks the destination Unity layer per SDK mask:

SDK mask Unity layer Effect on player capsule
CL_PHYSICS | CL_POINTER (Default mask) Default Solid (player blocks)
CL_PHYSICS alone CharacterOnly Solid
CL_POINTER alone OnPointerEvent Pass-through
CL_PLAYER and/or CL_MAIN_PLAYER (no other bits) SDKAvatarHit Pass-through; only trigger areas / raycasts targeting avatar bits detect it
Any mask containing CL_PHYSICS (mixed) CharacterOnly (physics wins) Solid
Custom layers SDKCustomLayer Pass-through

SDKAvatarHit is matrix-configured to only overlap with SDKEntityTriggerArea and SDKAvatarTriggerArea, so the player capsule walks straight through avatar-tagged scene meshes while trigger areas and avatar-targeting raycasts still pick them up.

Migration note for scenes pre-dating this rule. A scene that previously used CL_PHYSICS raycasts to detect the local player (e.g. line-of-sight checks) must now OR CL_MAIN_PLAYER (or CL_PLAYER) into the mask. Walls-and-floors checks that don't want to detect the player keep working unchanged.

"Perfect implementation" checks before shipping

Especially for big implementations:

  • Systems separation if system is too big
  • If using Promises: need TryGet() or can just get Result?
  • Use ListPool or ComponentPool when possible
  • Less "intention" components as possible, try to infer from existence/lack of component on Entity
  • Use UpdateGroup and Systems Update Order (UpdateBefore & UpdateAfter attributes)
  • Make sure scene feature doesn't run when it's not the current scene
  • Test coverage if possible
  • Allocations optimization

4. Create SDK example test-scene

This step is important because it allows creators (like the Content Team itself) to easily test the new component and know how it should be used, when the official documentation may still be pending.

  1. Clone the sdk7-test-scenes repo
  2. Create your test scene base
    • Duplicate one of the existent test scenes that exist in /scenes
    • Change the new folder name to your new test scene coordinates AND name (same format as the rest)
    • Update the README.md file
    • Update the package.json file -> "name" property
    • Update the scene.json file -> "name" property + "title" property + "base" and "parcels" coordinate properties
  3. Go to the root of the repo and run npm i and then npm run check-parcels to confirm there is no coordinates conflict between your new test scene and a previously existent one (that also updates the dcl-workspace.json file)
  4. Return to the root of your test scene and install an SDK version that contains the new component, for example if you are still working on an SDK branch with the component, the package is the one exposed in the js-sdk-toolchain PR, like:
npm install "https://sdk-team-cdn.decentraland.org/@dcl/js-sdk-toolchain/branch/feat/trigger-area-components/dcl-sdk-7.10.1-17412673762.commit-72cc7b1.tgz"
  • After that, your IDE should recognize the new component when imported from '@dcl/sdk/ecs' and you should be able to use it in your code without compilation errors.
  • NOTE: If the change you did to the protocol/sdk involves changing the API for global functions you may need to use the --save parameter when installing the test package like: npm install --save "https://sdk-team-cdn.decentraland.org/@dcl/js-sdk-toolchain/branch/feat/move-player-to-duration-field/dcl-sdk-7.17.1-20757258908.commit-16f1a78.tgz"
  1. Run your scene to confirm there are no compilation errors (npm run start -- --explorer-alpha)
  2. Code your test scene making use of the new component (index.ts and whatever extra scripts you need)
  3. Make sure to remove unneeded files like /models, etc. if your scene doesn't use them (/images/scene-thumbnail.png must be kept)
  4. Create a branch in the test scenes repo and push your new scene to create the corresponding PR in that repo, you are responsible for merging it