Segmentation helps isolate an area to apply customized textures and effects to it. The provided segmentation textures include portrait, hair, head, hand, and clothing. The textures also include sky, ground, and buildings for exploring.
Download the files below to follow along:
Add Segmentation Object: Portrait / Hair / Hand / Clothing Segmentation
Go to the Hierarchy panel and click the Add button [+] > Segmentation. Select the segmentation type you would like to add to your effect.
If there is no 2D Camera and Canvas in the Hierarchy panel, a 2D Background Camera object and a Background Canvas object will be automatically added with a Portrait/Hair/Hand/Clothing Segmentation object into the Hierarchy panel.
2D Background Camera object and Background Canvas object are set to Layer 2 to separate 2D segmentation image rendering from the default 3D camera rendering. To learn more about render layer and render order, see Image.
A Portrait Segmentation with Portrait and Background objects will show up in the Hierarchy panel whenever you add Portrait Segmentation:
- Portrait Segmentation is a container object with a 2D Transform.
- Portrait is essentially an image. The Image component of this object has been automatically assigned with the Portrait Segmentation Texture, which you can find in the Assets Panel.
- Background is essentially an image. The Image component of this object has been automatically assigned with Background Texture. You can make it animated by adding an Animation Sequence component.
A Hair / Hand / Clothing Segmentation with a Hair / Hand / Clothing Color object will show up in the Hierarchy panel whenever you add Hair, Hand, or Clothing Segmentation:
- Hair / Hand / Clothing Segmentation is a container with a 2D Transform.
- Hair / Hand / Clothing Color is an image object. If it is hair segmentation, the texture of the image is Hair Segmentation Texture. If it is hand segmentation, the texture of the image is Hand Segmentation Texture. If it is clothing segmentation, the texture of the image is Cloth Segmentation Texture. You can find these textures in the Assets panel.
- You can easily apply a custom texture to your hand/hair/clothing segmentation with this setup.
Additional Segmentation Types: Head / Sky / Ground / Building
When adding Segmentation to the scene, segmentation texture will be added to the Assets panel automatically. You may notice that, in addition to portrait / hair / hand / clothing segmentation, you can also do Head / Sky / Ground / Building Segmentation by configuring the Segmentation Texture‘s Type.
See Segmentation Texture to see how you can create a new Segmentation Texture and for more property details.
Demo Effect: Hand Segmentation
In this section, you’ll learn how to create a hand shadow effect.
Add a Hand Segmentation object by clicking the Add button [+] > Segmentation > Hand Segmentation in the Hierarchy panel.
Add an Image by clicking the Add button [+] > 2D > Image in the Hierarchy panel.This will be the background image. Remove the canvas and 2D camera that were added with the image. You only need one 2D camera and one canvas in the scene. Drag the Image under the Hand Color object so that it will render behind the hand segmentation. Rename it to Background.
Replace the texture of this Background image to background. Adjust its size in the Scene view to fill the screen.
Select Hand Segmentation Texture from the Assets panel. On the Inspector panel, change its Cutout Texture to the black hand texture you prepared. Now feel free to try it on your hand.
Demo Effect: Animated Sky Segmentation
In this section, you’ll learn how to create an animated sky effect. Feel free to download the attached project.
Go to the Hierarchy panel, click the Add button [+] > Segmentation > Portrait Segmentation.
Select the Portrait Segmentation Texture in the Assets panel. Navigate to the Inspector panel. Change the Type of the Portrait Segmentation Texture from Portrait to Sky.
Now, you’ll see the sky part gets cut out and the building areas are covered by the background image. To apply the background image to the sky area, you need to check Invert Mask.
To replace the background image with your custom animation, you need to add an Animation Sequence to the Background object first. Import your animation by clicking on the Add button [+] > Import > Texture Sequence in the Assets panel. See Texture Sequence to learn more about the asset requirements. Drag the imported animation to the Texture Sequence slot under Animation Sequence. Now, you should see the animation show up in the sky area.
Demo Effect: Add 3D Objects in Front of or Behind the Segmentation
You can add 3D objects in front of or behind the segmentation layer by changing the render order of the cameras or by manipulating the layers of each object.
Go to Hierarchy panel, then click the Add button [+] > 3D > Sphere. You should see that the Sphere is rendered on top of the Portrait Segmentation image and the Background image.
Render the Segmentation and Background Images Separately
To show the sphere between the Portrait image and the Background image, we need to have two 2D cameras to render these two images separately. Then, we can render the sphere in the middle.
First, go to the Hierarchy panel and duplicate Background Canvas with its child objects and 2D Background Camera. Rename the duplicate canvas to Segmentation Canvas and rename the duplicate camera to 2D Segmentation Camera.
Afterwards, disable the Portrait object from the original canvas and disable the Background object for Segmentation Canvas.
Change Render Layer
Click on Segmentation Canvas to open the Inspector panel and change the Layer to Layer 3.
Click on 2D Segmentation Camera and change the Layer to Layer 3 in order to render the Segmentation Canvas.
Make sure that the Sphere you added is at the same layer as the Camera.
Change Render Order
Make sure you update the Camera’s Render Order to a number that is between the Render Order of the 2D Background Camera and the 2D Segmentation Camera.
Go to 2D Segmentation Camera’s Render Order > Camera’s Render Order > 2D Background Camera’s Render Order so that the Segmentation image will render on top and then the 3D object, the back will be the background image.
Check Auto Sorting
Ensure that the Auto Sorting for the object has been checked to guarantee that the sorting order will follow the Hierarchy top-down order. You can also manually change the sorting order to manipulate the render order instead of enabling Auto Sorting.
For detailed information about Sorting orders, Layers, and Render Order, see Camera and Use multiple cameras.
Once you have updated the Render Order, you will see that the sphere now moves in between the segmentation and background canvases.
Performance on Device
Segmentation has a very high performance impact on iOS devices. We suggest not combining Segmentation with other features as doing so will cause low FPS and may subsequently lead to not passing QA.