How to visualize a point cloud in RealityKit on visionOS?

I attempted to use a LowLevelMesh and I referenced this sample.. I tried to change the .topology of the LowLevelMesh.Partin the initializeMeshParts function to .point. Referencing the documentation for MTLPrimitiveType.point states that "The vertex shader must provide [[point_size]], or the point size is undefined."

However, in this sample, the vertex shader function is not defined. What would be the best way to implement this simply?

Hey @ericD_TRI

Looks like you're on the right track—LowLevelMesh is the way to go for your use case, and you can actually use .point topology without writing a vertex shader yourself (since built-in materials as well as Shader Graph materials handle this for you)!

For example, the following code positions 1 million points randomly within a volume and renders them using low-level mesh:

struct PointCloudVertex {
    let position: SIMD3<Float>
    let color: SIMD3<Float>
}

extension PointCloudVertex {
    static var vertexAttributes: [LowLevelMesh.Attribute] = [
        LowLevelMesh.Attribute(semantic: .position, format: .float3, offset: MemoryLayout<Self>.offset(of: \.position)!),
        LowLevelMesh.Attribute(semantic: .color, format: .float3, offset: MemoryLayout<Self>.offset(of: \.color)!)
    ]

    static var vertexLayouts: [LowLevelMesh.Layout] = [
        LowLevelMesh.Layout(bufferIndex: 0, bufferStride: MemoryLayout<Self>.stride)
    ]

    static var descriptor: LowLevelMesh.Descriptor {
        var descriptor = LowLevelMesh.Descriptor()
        descriptor.vertexAttributes = PointCloudVertex.vertexAttributes
        descriptor.vertexLayouts = PointCloudVertex.vertexLayouts
        descriptor.indexType = .uint32
        return descriptor
    }
}

struct ContentView: View {
    var body: some View {
        RealityView { content in
            
            // Create a list of random point positions.
            let pointCount = 1000000
            var pointPositions: [SIMD3<Float>] = []
            for _ in 0..<pointCount {
                pointPositions.append(SIMD3<Float>.random(in: -1...1))
            }
            
            // Create the mesh descriptor.
            var meshDescriptor = PointCloudVertex.descriptor
            meshDescriptor.vertexCapacity = pointPositions.count
            meshDescriptor.indexCapacity = pointPositions.count
            
            // Create the low-level mesh.
            guard let lowLevelMesh = try? LowLevelMesh(descriptor: meshDescriptor) else {
                print("Failed to create low-level mesh.")
                return
            }

            // Populate the mesh's vertex buffer.
            lowLevelMesh.withUnsafeMutableBytes(bufferIndex: 0) { rawBytes in
                let vertices = rawBytes.bindMemory(to: PointCloudVertex.self)
                for i in 0..<pointPositions.count {
                    vertices[i] = PointCloudVertex(
                        position: pointPositions[i],
                        color: pointPositions[i]  // Color the points based on their positions. Or, if your points already have intrinsic colors associated with them, set that here instead.
                    )
                }
            }

            // Populate the mesh's index buffer.
            lowLevelMesh.withUnsafeMutableIndices { rawIndices in
                let indices = rawIndices.bindMemory(to: UInt32.self)
                for i in 0..<pointPositions.count {
                    indices[i] = UInt32(i)
                }
            }

            // Set the index count, topology, and bounds of the mesh.
            let meshBounds = BoundingBox(min: [-1, -1, -1], max: [1, 1, 1])
            lowLevelMesh.parts.replaceAll([
                LowLevelMesh.Part(
                    indexCount: pointPositions.count,
                    topology: .point,
                    bounds: meshBounds
                )
            ])
            
            // Create a mesh resource from the low-level mesh.
            guard let mesh = try? await MeshResource(from: lowLevelMesh) else {
                print("Failed to create mesh from low-level mesh.")
                return
            }
            
            // Add the point cloud to the scene.
            let pointCloud = Entity(components: [ModelComponent(mesh: mesh, materials: [SimpleMaterial()])])
            content.add(pointCloud)
        }
    }
}

If you want to visualize the colors of the points, you will also need to create a custom Shader Graph material in Reality Composer Pro. In that material, add a Geometry Color node and connect it to the Diffuse Color input of the Preview Surface node.

Then, load that material and apply it to the point cloud entity by replacing the code below the // Add the point cloud to the scene. line above with the following:

// Load the material.
guard let material = try? await ShaderGraphMaterial(named: "/Root/PointMaterial", from: "Materials/PointMaterial", in: realityKitContentBundle) else {
    return
}

// Add the point cloud to the scene with the material applied.
let pointCloud = Entity(components: [ModelComponent(mesh: mesh, materials: [material])])
content.add(pointCloud)

Here, I had created a scene with the name "PointMaterial" in the "Materials" folder in RCP, and added my custom Shader Graph material as a descendant to the "Root" entity of that scene with the name "PointMaterial" as well. You may need to adjust the named: and from: parameters of the ShaderGraphMaterial initializer depending on the name and location of your material.

You should now see the color of the points form a gradient based on their positions. If you have a list of color data as well as position data, you could then adjust the code above to display that instead.

Let me know if you have any further questions!