Gatsby RSS feed

Images in WebGL and react-three-fiber

Thu, 11 Jun 2020 00:00:00 GMT

The image effects you can do with WebGL is what gets me excited about this sort of stuff. The trouble is that there’s a bit of a learning curve and it’s certainly not easy. The good thing is that once you’ve got it figured out and set up, you can re-use your base and improve/adjust things as you learn.

I’m going to walk through two ways you can use images in your work. The first being a simple method that will provide a foundation for understanding how to move images from the DOM into WebGL-land; and the second will build off the first method and introduce the concept of shaders - the magic ingredient!

The Situation

The first thing that you need to be aware of is that everything inside the canvas isn’t part of the regular flow of the DOM. This means that you’ll need to be sizing and positioning things yourself but this isn’t a bad thing. In fact it fits in perfectly with the idea of progressive-enhancement.

What we’ll do is render our images in the DOM as we would normally. Then we’ll grab them all and pass them into our canvas. Doing things this way means we know:

The image has loaded
The image dimensions
The image position in the DOM

It also means that the images remain fully-accessible on the page and that we lose nothing of value if for some reason the WebGL doesn’t load or someone has their JavaScript turned off.

Collecting the Images

Let’s use some react context and a custom component to scoop up all of these images.

Context

For context (heh) I follow the pattern explained by Kent C. Dodds.

// src/webgl/context.js
import React, { useState, createContext, useContext } from 'react'
const WebGLStateContext = createContext()
const WebGLDispatchContext = createContext()
export const WebGLProvider = ({ children }) => {
  const [state, dispatch] = useState([])
  return (
    <WebGLStateContext.Provider value={state}>
      <WebGLDispatchContext.Provider value={dispatch}>
        {children}
      </WebGLDispatchContext.Provider>
    </WebGLStateContext.Provider>
  )
}
export function useWebGLState() {
  const context = useContext(WebGLStateContext)
  if (context === undefined) {
    throw new Error('useWebGLState must be used within a WebGLProvider')
  }
  return context
}
export function useWebGLDispatch() {
  const context = useContext(WebGLDispatchContext)
  if (context === undefined) {
    throw new Error('useWebGLDispatch must be used within an WebGLProvider')
  }
  return context
}

What we’ve done here is setup a react context that will hold all of the images that we’ll then use in our WebGL code.

Then to hook it up we’ll need to add <WebGLProvider /> somewhere, if you’re not sure where is best then I’d recommend the app root.

Next step is putting together a little component to grab any images we want to use. To keeps things simple I’m going to add this to our context file we’ve just setup.

// src/webgl/context.js
import React, { useRef, useState } from 'react'
export const WebGLImage = props => {
  const ref = useRef()
  const dispatch = useWebGLDispatch()
  const [loaded, setLoaded] = useState(false)
  const handleLoad = () => {
    setLoaded(true)
    dispatch(images => [...images, { data: ref.current }])
  }
  return (
    <img
      alt=""
      {...props}
      ref={ref}
      onLoad={handleLoad}
      style={{
        opacity: loaded ? 0 : 1,
      }}
    />
  )
}

Setting up the Canvas

// src/webgl/canvas.js
import React, { Suspense, useMemo } from 'react'
import { Canvas, useLoader, useThree } from 'react-three-fiber'
import { TextureLoader, LinearFilter, ClampToEdgeWrapping } from 'three'
import { useWebGLState } from './context'
// Next step
function Image() { ... }
function WebGLCanvas() {
  const state = useWebGLState()
  return (
    <Canvas
      orthographic
      style={{
        height: '100vh',
        position: 'fixed',
        top: 0,
        right: 0,
        bottom: 0,
        left: 0,
        zIndex: 1,
        pointerEvents: 'none',
        transform: 'translateZ(0)',
      }}
    >
      <Suspense fallback={null}>
        {state.map((image, i) => (
          <Image key={image.data.src} image={image} />
        ))}
      </Suspense>
    </Canvas>
  )
}
export default WebGLCanvas

Lets go over what’s happening here:

We switch over to an orthographic camera
We style the canvas to fill the viewport and set it to a fixed position
Switching off pointer events means it doesn’t get in the way of mouse events (like the context menu)
Setting a transform tells the browser to move rendering the canvas element to the GPU
<Suspense /> is being used to deal with image loading
We loop over all of the images we’ve added to our context and render them in a new <Image /> component

Rendering the Images

There’s a lot to unpack in this next section so I’ve added comments to the code example, so let’s dig in.

const Image = ({ image }) => {
  // The first step is to load the image into a texture that we can use in WebGL
  const texture = useLoader(TextureLoader, image.data.src)
  // Then we want to get the viewport from ThreeJS so we can do some calculations later
  const { viewport } = useThree()
  // We need to apply some corrections to the texture we've just made
  useMemo(() => {
    texture.generateMipmaps = false
    texture.wrapS = texture.wrapT = ClampToEdgeWrapping
    texture.minFilter = LinearFilter
    texture.needsUpdate = true
  }, [
    texture.generateMipmaps,
    texture.wrapS,
    texture.wrapT,
    texture.minFilter,
    texture.needsUpdate,
  ])
  // Here we grab the size and position of the image from the DOM
  const { width, height, top, left } = image.data.getBoundingClientRect()
  return (
    <mesh
      // We convert the width and height to relative viewport units
      scale={[
        (width / window.innerWidth) * viewport.width,
        (height / window.innerHeight) * viewport.height,
        1,
      ]}
      // We convert the x and y positions to relative viewport units
      position={[
        ((width / window.innerWidth) * viewport.width) / 2 -
          viewport.width / 2 +
          (left / window.innerWidth) * viewport.width,
        0 -
          ((height / window.innerHeight) * viewport.height) / 2 +
          viewport.height / 2 -
          (top / window.innerHeight) * viewport.height,
        0,
      ]}
    >
      {/* We're use a simple plane geometry */}
      {/* think of it like a piece of paper as a 3d shape */}
      <planeBufferGeometry attach="geometry" />
      {/* Finally we map the texture to a material */}
      {/* or in other terms, put the image on the shape */}
      <meshBasicMaterial attach="material" map={texture} />
    </mesh>
  )
}

I chose this image because it reflects how I feel about doing this sort of math.

For anyone who knows of a better way please do let me know. Otherwise for anyone who is interested in the thought process behind the maths:

To get the scale

Divide the width of the image by the width of the window to get a percentage
Multiply that by the width of the canvas viewport to get the relative width within the canvas

To get the x value

Perform the same calculation as above to get the relative width within the canvas
Divide that by 2 to move the image half of its width to the right, this aligns the left side of the image with the center of the canvas
Subtract half of the canvas width to move the image to the left edge of the canvas
Perform a calculation to convert the original left value into a relative canvas value
Add that new relative value to move the image across to its final position

To get the y value

You have to do the opposite of the last section to get a matching relative top position

Subtract the following calculation from 0 because we’re moving in an opposite direction
Perform the same calculation above to get the relative height within the canvas
Divide that by 2 to move the image down by half its height, this aligns the top side of the image with the center of the canvas
Add half of the canvas width to move the image to the top edge of the canvas
Perform a calculation to convert the original top value into a relative canvas value
Subtract that new relative value to move the image down

Introducing Shaders

A <meshBasicMaterial /> is a perfectly fine thing to use, but unless you’ve got some grand ideas then for the purpose of rendering images you might be better off without WebGL. The key to the fun effects you’re looking for belong in shader-land, for this we’ll be using a <shaderMaterial /> instead.

The first thing we’re going to do is create a new file:

// src/webgl/image.js
import * as THREE from 'three'
import { extend } from 'react-three-fiber'
export default class Image extends THREE.ShaderMaterial {
  constructor() {
    super({
      uniforms: {
        texture: { type: 't', value: undefined },
      },
      vertexShader: `
        varying vec2 vUv;
        void main() {
          vUv = uv;
          gl_Position = projectionMatrix * modelViewMatrix * vec4( position, 1.0 );
        }`,
      fragmentShader: `
        varying vec2 vUv;
        uniform sampler2D texture;
        void main() {
          vec2 uv = vUv;
          vec4 texture = texture2D(texture, uv);
          gl_FragColor = texture;
        }`,
    })
  }
  // These let us set the texture uniform with a react prop
  get texture() {
    return this.uniforms.texture.value
  }
  set texture(v) {
    this.uniforms.texture.value = v
  }
}
// register an element in r3f as <image />
extend({ Image })

There are 3 main points to understand in this new file:

uniforms - these are variables that we share with the shader code
vertexShader - this shader handles the positioning of the pixels being rendered
fragmentShader - this shader handles the colour of the pixels being rendered

Following these points we’re passing a texture (an image) into our shaders. The shaders then get the positioning and colours of each pixel (the uv value) and then draw it to the screen.

For more information on shaders I’d highly recommend The Book of Shaders. Otherwise you can go find shaders other people have written and copy/jury-rig them into place.

The next step is to then import this new file into our canvas.js

// src/webgl/canvas.js
import React, { Suspense, useMemo } from 'react'
import { Canvas, useLoader, useThree } from 'react-three-fiber'
import { TextureLoader, LinearFilter, ClampToEdgeWrapping } from 'three'
import { useWebGLState } from './context'
import './image'

Then we switch out the <meshBasicMaterial /> with our new <image /> component!

<image attach="material" texture={texture} />

Here’s an embed of the finished code:

And there we have it. Now we’re free to start adding some cool effects!

WebGL Shader Glossary

Fri, 05 Jun 2020 00:00:00 GMT

A personal reference for WebGL shader terms. Feel free to get in touch if anything isn’t clear or if there’s something you would like to add.

Terms

Normalizing: Converting a value to be between 0.0 and 1.0. You can normalize the FragCoord by dividing it with by the canvas resolution

Variables

Note: when I say “between pixels” it’s because a shader will perform its calculations on every individual pixel.

Uniform: The same between pixels
Varying: Can be different between pixels
FragColor: The colour of the pixel being computed
FragCoord: The x and y value of the pixel being computed

An intro to WebGL and react-three-fiber

Wed, 03 Jun 2020 00:00:00 GMT

This is a series of articles that follow my journey with WebGL and react-three-fiber. They serve as both a reference for myself and a resource for anyone who is finding similar struggles in the world of WebGL.

Why WebGL

For anyone that’s not sure what WebGL really is, it’s a way of writing interactive code on the web (using a canvas) that’s somewhat similar to game development.

You have access to 3D models, materials, and shaders.

A simple way of explaining these parts would be:

You have a 3D model which is a shape
You apply a material to that model in order to give it some colour
You can also choose to add some shaders. These are the brains of the operation, they dictate the rules and let you do all of the cool stuff.

The bit that I’m interested in is learning how to use all of this to enhance the experiences I build. If you go to any web awards it’s almost guaranteed that most of the winning submissions have some form of WebGL woven through; and these are the best examples that show the limit of what you can do is up to your imagination.

Why react-three-fiber

I’m using react-three-fiber because most of my work at the moment revolves around react (across any platform), however it’s not much different than doing it in three.js so the skills are transferable. I’ve decided to go down this route rather than starting from the bottom with raw WebGL/OpenGL because I prefer fast, easy results that keep me excited and having fun early on. Once I’ve gotten the hang of things I am then free to dive into the vanilla code at my leisure. A solid foundation of experience will make the vanilla code more approachable.

Getting set up

Start off getting a base development environment set up that matches how you like to code. For ease, I’m going to be using codesandbox to handle the environment. This means I’m up and ready with one click and I can publicly link to my work as I go.

The dependencies that you’ll need are:

react-three-fiber

This library handles a lot of the boilerplate of setting up a three.js project and bringing it into React means you can make use of both state and component goodness.

To start with, here’s the finished codesandbox so you can see what we’re making and have a tinker.

The basic explanation of how to write react-three-fiber code instead of three.js, is that everything is in camelCase. If we use something like pointLight as an example then things will look a little like this:

// three.js
var light = new THREE.PointLight(0xff0000, 1, 100)
light.position.set(50, 50, 50)

Becomes

// react-three-fiber
<pointLight position={[50, 50, 50]} />

With this in mind, here is some basic three.js code that sets up a box. The steps are fairly logical in the sense that: we setup a box geometry (a box shape), we setup a material that has some colour, then we combine them both in a both in a mesh. This tells the mesh what shape to take (the geometry) and what to look like (the material).

// three.js
var geometry = new THREE.BoxBufferGeometry(1, 1, 1)
var material = new THREE.MeshStandardMaterial({ color: 0xffff00 })
var mesh = new THREE.Mesh(geometry, material)

In react-three-fiber it would look like this:

// react-three-fiber
<mesh>
  <boxBufferGeometry attach="geometry" args={[1, 1, 1]} />
  <meshStandardMaterial attach="material" color="orange" />
</mesh>

That’s the basics for setting up a simple 3D box and colouring it orange. If we wanted to make this re-usable we could return this from a function and have it be part of a normal React component. For the time being we’ll keep it as it is. Now in order to actually render it to the screen we need to have a canvas and some basic lights.

<Canvas>
  <ambientLight />
  <pointLight position={[10, 0, 0]} />
  <mesh rotation={[1, 0.5, 1]} scale={[3, 3, 3]}>
    <boxBufferGeometry attach="geometry" args={[1, 1, 1]} />
    <meshStandardMaterial attach="material" color="orange" />
  </mesh>
</Canvas>

If we walk through this step by step:

<Canvas /> sets up the HTML <canvas /> and handles the three.js boilerplate with a default camera rig.
<ambientLight /> provides generic lighting to the scene, like flicking a lightswitch.
<pointLight position={[10, 0, 0]} /> adds some directional lighting so we can see that our box is 3D. The position attribute takes an array of 3 numbers - these are the x, y, and z values respectively.

You might also notice that I’ve added some attributes to our <mesh />, this is just to make it a bit bigger and give it a bit of rotation so you can really see how the point light highlights that it’s a 3D model.

This is all you need!