Skip to main content
TikTok Effect House Platform Logo
Download
লগইন করুন
Tracking and detection

How to use Segmentation

Segmentation helps isolate a selected body part in order to apply customized textures and effects to it. The provided segmentation textures include portrait, hair, head, hand, and clothing. The textures also include sky, ground, and buildings for exploring.

↓ demo_handseg.zip

↓ AnimatedSkySegmentation.zip

↓ Add3DobjectToSegmentation.zip

Add Segmentation Object: Portrait / Hair / Hand / Cloth Segmentation

Go to the Hierarchy panel and click + Add object > Segmentation. Select the segmentation type you would like to add to your effect.

If there is no 2D Camera and Canvas in the Hierarchy panel,a 2D Background Camera object and a Background Canvas object will be automatically added with a Portrait/Hair/Hand/Cloth Segmentation object into the Hierarchy panel.

2D Background Camera object and Background Canvas object are set to Layer 2 to separate 2D segmentation image rendering from the default 3D camera rendering. To learn more about render layer and render order, see Image.

unaligned-image-0
unaligned-image-1

A Portrait Segmentation with Portrait and Background objects will show up in the Hierarchy panel whenever you add Portrait Segmentation:

  • Portrait Segmentation is a container object with a 2D Transform.
  • Portrait is essentially an image. The Image component of this object has been automatically assigned with the Portrait Segmentation Texture, which you can find in the Assets Panel.
  • Background is essentially an image. The Image component of this object has been automatically assigned with Background Texture. You can make it animated by adding an Animation Sequence component.
unaligned-image-0
unaligned-image-1
unaligned-image-0
unaligned-image-1

A Hair / Hand / Cloth Segmentation with a Hair / Hand / Cloth Color object will show up in the Hierarchy panel whenever you add Hair, Hand, or Cloth Segmentation:

  • Hair / Hand / Cloth Segmentation is a container with a 2D Transform.
  • Hair / Hand / Cloth Color is an image object. If it is hair segmentation, the texture of the image is Hair Segmentation Texture. If it is hand segmentation, the texture of the image is Hand Segmentation Texture. If it is cloth segmentation, the texture of the image is Cloth Segmentation Texture. You can find these textures in the Assets panel.
  • You can easily apply a custom texture to your hand/hair/cloth segmentation with this setup.

Additional Segmentation Types: Head / Sky / Ground / Building

When adding Segmentation to the scene, segmentation texture will be added to the Assets panel automatically. You may notice that, in addition to portrait / hair / hand / cloth segmentation, you can also do Head / Sky / Ground / Building Segmentation by configuring the Segmentation Texture‘s Type.

See Segmentation Texture to see how you can create a new Segmentation Texture and for more property details.

Demo Effect: Hand Segmentation

In this section, you’ll learn how to create a hand shadow effect.

demo_handseg.zip

Add a Hand Segmentation object by clicking + Add object > Segmentation > Hand Segmentation.

Add an Image by clicking + Add object > 2D > Image. This will be the background image. Remove the canvas and 2d camera that were added with the image. You only need one 2D camera and one canvas in the scene. Drag the Image under the Hand Color object so that it will render behind the hand segmentation. Rename it to Background.

Replace the texture of this Background image to background. Adjust its size in the Scene view to fill the screen.

Select Hand Segmentation Texture from the Assets panel. On the Inspector panel, change its Cutout Texture to the black hand texture you prepared. Now feel free to try it on your hand.

Demo Effect: Animated Sky Segmentation

In this section, you’ll learn how to create an animated sky effect. Feel free to download the attached project.

↓ AnimatedSkySegmentation.zip

Go to the Hierarchy panel, click + Add object > Segmentation > Portrait Segmentation.

Select the Portrait Segmentation Texture in the Assets panel. Navigate to the Inspector panel. Change the Type of the Portrait Segmentation Texture from Portrait to Sky.

Now, you’ll see the sky part gets cut out and the building areas are covered by the background image. To apply the background image to the sky area, you need to check Invert Mask.

To replace the background image with your custom animation, you need to add an Animation Sequence to the Background object first. Import your animation by clicking on + Add Asset > Import > Texture Sequence. See Texture Sequence to learn more about the asset requirements. Drag the imported animation to the Texture Sequence slot under Animation Sequence. Now, you should see the animation show up in the sky area.

Demo Effect: Add 3D objects in front of or behind the segmentation

↓ Add3DobjectToSegmentation.zip

You can add 3D objects in front of or behind the segmentation layer by changing the render order of the cameras or by manipulating the layers of each object.

Go to Hierarchy panel, then choose + Add object > 3D > Sphere. You should see that the Sphere is rendered on top of the Portrait Segmentation image and the Background image.

Render the segmentation and background images separately

To show the sphere between the Portrait image and the Background image, we need to have two 2D cameras to render these two images separately. Then, we can render the sphere in the middle.

First, go to the Hierarchy panel and duplicate Background Canvas with its child objects and 2D Background Camera. Rename the duplicate canvas to Segmentation Canvas and rename the duplicate camera to2D Segmentation Camera.

Afterwards, disable the Portrait object from the original canvas and disable the Background object for Segmentation Canvas.

Change Render Layer

Click on Segmentation Canvas to open the Inspector panel and change the Layer to Layer 3.

Click on 2D Segmentation Camera and change the Layer to Layer 3 in order to render the Segmentation Canvas.

Tip
Tip

Make sure that the Sphere you added is at the same layer as the Camera.

Change Render Order

Make sure you update the Camera’s Render Order to a number that is between the Render Order of the 2D Background Camera and the 2D Segmentation Camera.

2D Segmentation Camera’s Render Order > Camera’s Render Order> 2D Background Camera’s Render Order so that the Segmentation image will render on top and then the 3D object, the back will be the background image.

Check Auto Sorting

Tip
Tip

Ensure that the Auto Sorting for the object has been checked to guarantee that the sorting order will follow the Hierarchy top-down order. You can also manually change the sorting order to manipulate the render order instead of enabling Auto Sorting.

For detailed information about Sorting orders, Layers, and Render Order, see Camera and Use multiple cameras.

Demo

Once you have updated the Render Order, you will see that the sphere now moves in between the segmentation and background canvases.

Performance on Device

Segmentation has a very high performance impact on iOS devices. We suggest not combining Segmentation with other features as doing so will cause low FPS and may subsequently lead to not passing QA.