Learn how to add and manipulate images and videos in your p5.js sketch!

Images and Videos

If you've ever web-surfed with poor internet connection—you know firsthand that images can take a while to show up, even if other parts of the page load just fine.

This same behavior applies to p5.js sketches. Loading images takes time, and p5.js will typically move onto other code before an image fully loads.

In the last exercise, this was okay because we drew our image in the infinitely looping `draw()` function. Even if the image wasn't loaded when the `image()` function was first called, it eventually could be drawn to the canvas.

But sometimes, it's important that images are completely loaded before we attempt to draw them. For example, check out what happens when we try to load and draw the image within the `setup()` function:

![A diagram that demonstrates what happens when you load and try to draw an image in the setup() function. The image never ends up drawing to the canvas.](https://static-assets.codecademy.com/Courses/Learn-p5/media/noPreload.gif)

Here, the image never shows up! By the time the `image()` function is called, the image isn't fully loaded yet. The `setup()` function runs only once, so the image is not eventually drawn.

To solve problems like this, p5.js includes the `preload()` function. Like the `setup()` and `draw()` functions, it's a special function to hold specific kinds of code for our p5.js sketch. Specifically, the `preload()` function is meant for code that loads external assets, like the `loadImage()` function.

```js
let img;

function preload(){
	img = loadImage('myImage.png');
}
```

The p5.js library calls the `preload()` function before the `setup()` and `draw()` functions, which run only once all assets in `preload()` are entirely loaded.

```js
let img;

function preload(){
  //This runs first.
  img = loadImage('myImage.png');

function setup(){
  //This runs once when all assets in preload() are completely loaded.
}
function draw(){
  //This loops infinitely after setup() finishes.
}
```

Let's take a look at our previous scenario, now with the `preload()` function!

![A diagram explaining the preload function. It walks through how p5.js will wait to load any images in preload() completely before moving onto the setup and draw functions. The image successfully draws to the canvas.](https://static-assets.codecademy.com/Courses/Learn-p5/media/preload.gif)

Our (very adorable) image now draws to the canvas! This is because p5.js waits to run the code in the `setup()` function only after the image is fully loaded.

The Preload Function

Now that we can add videos to the canvas let's take a look at some extra options p5.js gives us to control video options and playback. These options also build off of HTML and JavaScript's existing support for video playback.

We already learned how to use the `.play()` method to play a video. In actuality, there is another way to get a video to play: the `.loop()` method. Whereas the `.play()` method only plays a video once, the `.loop()` method will play a video and loop it infinitely.

```js
video.loop();
```

Because of how these methods work behind-the-scenes in JavaScript, there are some quirks that can lead to common problems. One of them is that calling `.play()` or `.loop()` way too many times, such as in every iteration of the `draw()` loop, can cause errors. In general, it's best to call them in `setup()`, or call them in response to user input, like mouse clicks.

To stop a playing or looping video, use the `.stop()` method.

```js
video.stop();
```

The `.stop()` method stops the video and returns it to the starting frame of the video. To pause the video at the current frame—so that you can later resume it where you left off—use the `.pause()` method:

```js
video.pause();
```

To change the volume of the video's sound, use the `.volume()` method. It takes in a decimal number ranging from 0 to 1, with 0 being no volume and 1 being maximum volume.

```js
video.volume(0); // Mutes the sound
video.volume(1); // Plays sound at max volume
video.volume(0.5); //Plays sound at half volume
```

Video Options

In this lesson, we learned how to bring media into our p5.js Sketches with images and videos. We also learned how to play around with them through filters and pixel manipulation.

Let's take a moment to review what we covered:

- The `loadImage()` function is used to load an external image into your p5 Sketch. The function must be provided with the path to the image as an argument in string format.
- The `preload()` function is meant to hold any code for loading media that needs to be completely loaded before the `setup()` function and `draw()` loop run.
- The `image()` function draws an image to the canvas, taking in the given image element and the x and y location of where the image should be drawn.
 - Optionally, arguments for width, height, and crop dimensions can be given to the function.
- The `createVideo()` function creates an HTML video element that is displayed outside of the sketch. It takes in a single path to a video as a string.
- p5.js supports various methods that affect playback of videos, such as `.loop()`, `.play()`, `.stop()`, `.pause()`, and `.volume()`.
- Filters are a way to apply effects all-over an entire canvas or on an image element. Both the `filter()` function for canvases and `.filter()` method for images require the type of filter as an argument.
 - Certain filter types require an additional numerical argument. 
 - While filters can be used on images, they can not be applied to individual video elements.
- The `get()` function accesses the pixel color on the canvas at a specific x,y coordinate, as an array of four values representing the red, green, blue, and alpha values.
 - Alternately, the `get()` function can return a region of the canvas as an image element—when provided with two additional parameters for width and height, it returns the region starting at the x, y position, bounded by the width and height.
 - When provided with no arguments, it returns the entire canvas region as an image.
- The `set()` function sets a pixel at a given x, y coordinate to a color, which can be an array of four RGBA values, a single greyscale value, or a p5.js color element.
 - To reflect changes made with the `set()` function, you must call the `updatePixels()` function.
- The `pixels` array provides a faster way to read and modify pixels. It lists the R, G, B, A values for every pixel in an image or canvas in one single array.
- Before accessing the pixels array, you must load the pixels with the `loadPixels()` function.
- After making modifications to the pixels array, you must call `updatePixels()` to see those changes reflected.

Review

So far, we've learned to create dynamic p5.js sketches using only shapes and colors.

But, just like how you might add external images or videos onto a normal webpage, you can also use the p5.js library to incorporate external images or videos directly into the canvas!

By the end of this lesson, you will be able to:

- Load and draw external images into the canvas
- Display external videos in the canvas
- Add filters effects across images and the canvas
- Read and manipulate the pixels within the canvas, as well as in images and videos

Introduction to Images and Videos

Now that we can add images into the canvas let's learn how to add another major type of media: videos!

To create a p5.js video element, use the `createVideo()` function. It requires the video file's path in your working directory. Here, we'll use a video with the .mp4 format, which is a video file format that's well-supported across most browsers.

```js
let video;
video = createVideo('myVideo.mp4');
```

As with images, we can store video elements as variables. Typically, we'll do this in a place like the `preload()` function so that we can use it later on in our sketch.

To play a video, you can use the `.play()` method:

```js
video.play();
```

Leveraging HTML and JavaScript's existing support for videos, the `createVideo()` function displays an HTML video element on the page, but not within the sketch itself.

To display the video directly within the p5 sketch, we can reuse the `image()` function from before:

```js
image(video, 0, 0);
```

It may be confusing to use a function called 'image' to draw videos—but it makes sense when we think of a video as a sequence of images. When called, the `image()` function draws the current frame of the video on the canvas. When placed in the `draw()` loop, we can create the illusion of a video playing, frame-by-frame.

Like with drawing images, you can resize the drawn video by adding two more parameters:

```js
let drawnWidth = 100;
let drawnHeight = 100;
image(video, 0, 0, drawnWidth, drawnHeight);
```

To hide the original HTML video element, we can use the `.hide()` method:

```js
video.hide()
```

This makes the original video invisible—but it doesn't delete it! You'll still be able to hear the video's sound unless you turn off its volume. We'll get to that in the next exercise!

Note that the `.hide()` method will hide the video drawn on the canvas as well unless at any point you play the video. Until then, no video frames will draw to the canvas.



Videos

Before diving into more complex image manipulation techniques, let's look at what's happening under the hood with the `get()` and `set()` functions.

The `pixels` array is a special data structure that enables us to access and manipulate pixel data—whether in the canvas or in individual images or videos. The `get()` and `set()` functions are actually functions that make working with pixels array easier, but slower.

We normally think of an image as a two-dimensional matrix—an array of arrays—with each pixel having a red, green, blue, and alpha (RGBA) value.

The `pixels` array is a representation of all those pixels, flattened down to a single list. Starting from the first pixel and moving left-to-right by rows to the last pixel, `pixels` stores the RGBA values for each pixel into one, single array. The one-dimensional array format, while harder to work with, makes accessing the data fast.

The pixels array is also more like a snapshot of a canvas or image, rather than a source of truth for what's on the screen. This is where the `updatePixels()` function comes back in—after we modify the `pixels` array, it's how we indicate that we're done making changes, and we're ready to see it updated on the sketch.

In some cases, the `pixels` array will be affected by _pixel density_. The pixel density in a p5.js sketch depends on your computer monitor—many high-resolution monitors treat a single pixel on the screen as multiple smaller pixels, creating a sharper image—accordingly, the number of elements in the `pixels` array factors in that density.

A diagram that shows how pixels are structured in an image.

The Pixels Array

Adding photos and videos into the canvas is cool, but the real fun comes when we add effects to them through tools like filters.

Filters in p5.js work in two ways: across the entire canvas and applied to individual images.

To apply a filter across the canvas, use the `filter()` function. It requires the type of filter, which can be one of eight different types as defined by p5.js. For a complete list, visit the [p5.js reference on filters](https://p5js.org/reference/#/p5/filter).

For example, to add a `GRAY` filter across the canvas, use:

```js
filter(GRAY);
```

![A diagram showing what the GRAY filter looks like when applied to a canvas with a pink background and an image of a puppy. The entire canvas becomes grayscale.](https://static-assets.codecademy.com/Courses/Learn-p5/media/grayFilter.gif)

Certain filter types require an additional numerical argument. For example, the `POSTERIZE` filter, which reduces the number of colors in the image, requires a value between 2 and 255.

```js
filter(POSTERIZE, 3);
```

![A diagram showing what the POSTERIZE filter looks like when applied to a canvas with a pink background and an image of a puppy. The canvas becomes reduced to fewer colors.](https://static-assets.codecademy.com/Courses/Learn-p5/media/posterizeFilter.gif)

The `filter()` function applies the filter to everything drawn on the canvas before it's called—this lets us layer filters together, combining them with ones that were called previously.

![A diagram showing what the GRAY and POSTERIZE filter look like when used one after another. The canvas turns grayscale and is reduced to a few shades of gray.](https://static-assets.codecademy.com/Courses/Learn-p5/media/grayAndPosterizeFilter.gif)

It also lets us apply filters to certain parts of the canvas—for example, we can apply a filter to the entire canvas, then draw new elements that'll be unaffected.

![A diagram showing the grayscale filter applied to a canvas, but another colored image of a flower is drawn on top, unaffected by the filter.](https://static-assets.codecademy.com/Courses/Learn-p5/media/layersFilters.gif)

To apply filters across an individual p5 image element, use the `.filter()` method. You'll need to call this before drawing the image.

```js
//Invert colors in an image
img.filter(INVERT); 
image(img, 0, 0);
```

![A diagram showing the invert filter only being applied to one image on the canvas. The rest of the elements are unaffected.](https://static-assets.codecademy.com/Courses/Learn-p5/media/imageFilter.gif)

We can't, however, apply filters to individual video elements (though you could instead draw a video to the canvas, then add a filter to the entire canvas).

A weakness of the `filter()` function is its slow performance—especially when called frequently. Later, we'll learn how to achieve similar effects (and more!) with pixel manipulation.

Filters

As mentioned earlier, pixel density can affect how you read the `pixels` array. Since different monitors have different pixel densities, let's make things uniform by working with a pixel density of `1`. To do this, use the `pixelDensity()` function.

```js
function setup(){
  //Return the current pixel density
  pixelDensity(); 
  //Set the pixel density to 1
  pixelDensity(1); 
}
```

We could omit this step, but that'll make some of the math harder to work with later on!

One quirk of setting a different pixel density using the `pixelDensity()` function is that it erases everything you drew on the canvas before calling it. In most cases, it's a good idea to call it first thing in the `setup()` function, before any other code.

```js
function setup() {
  // Set pixel density first
  pixelDensity(1);
  // The rest of the code can follow:
  createCanvas(400, 400)
  background(220);
}
```

Before accessing the pixels array, there's one more step: loading the pixels with the `loadPixels()` function. `loadPixels()` loads the pixels in a canvas, populating the built-in pixels variable.

```js
//Load the canvas pixels
loadPixels();
//Print the canvas' pixels array
console.log(pixels); 
```

Just like with `get()` and `set()`, you can also access pixels for individual images or videos. To do this, use the `.loadPixels()` method, which will populate the `.pixels` property on the image or video element.

```js
//Load image pixels
img.loadPixels(); 
//Prints the image's pixels array
console.log(img.pixels); 
```

Note that when dealing with videos (or animated GIFs), it's better to call the `loadPixels()` function in the `draw()` loop&mdash;you likely want to work with the latest frame.

```js
function draw(){
  //Load video pixels of latest frame
  video.loadPixels(); 
  //Prints the video's pixels array
  console.log(video.pixels); 
}
```

Loading the data in the pixels array takes time—which can explain why the pixels array isn't just available by default. In fact, the `get()` and `set()` functions both load the pixels every single time they're called—which is why calling them a lot slows down the sketch significantly.

In contrast, accessing the entire `pixels` array requires loading pixels just once—which speeds up the process a lot and will allow us to do more wide-scale manipulations!

Still, keep in mind that working with the `pixels` array can potentially slow down your sketch, especially if you:
- Load and manipulate the `pixels` array of high-resolution images or videos
- Load and manipulate the `pixels` array in the `draw()` loop.
- Print the `pixels` array to the console (we did it as a demonstration above, but this can really affect performance!)

Doing any of these alone may not cause problems, but you can troubleshoot laggy p5.js sketches by being mindful of the above.

Loading the Pixels Array

We can also use p5.js to manipulate images and videos, down to the pixel. p5.js offers built-in functions to do this: `get()` and `set()`.

The `get()` function accesses the color of a specific pixel on the canvas. When given a pixel location, it returns the color of that pixel as an array of four numbers, representing the red, green, blue, and alpha (RGBA) values.

```js
let pixelColor = get(28, 35); // Returns [r, g, b, a], where r, g, b, a are values between 0 and 255
```

![A diagram that shows the get() function on a small photo of a puppy. It returns a pink color at coordinates (28, 35).](https://static-assets.codecademy.com/Courses/Learn-p5/media/get.png)

Alternatively, the `get()` function also can retrieve regions of the canvas, returning them as a p5.js image element.

```js
// Using get() to access a portion of canvas
let x = 0; let y = 0;
let w = 100; let h = 100;
let partOfCanvas = get(x, y, w, h); // Returns a region starting at (x, y) that's w-pixels wide and h-pixels high
```

When provided no arguments, it retrieves the entire canvas as a p5.js image element.

```js
// Using get() to access entire canvas
let entireCanvas = get();
```

The `set()` function sets a pixel at a given location on the canvas to a new color.

```js
let red = [255, 0, 0, 255];
set(28, 35, red); // Sets pixel at (28, 35) to red
```

![A diagram that shows the set() function on a small photo of a puppy. It changes the pixel at (28,35) to red.](https://static-assets.codecademy.com/Courses/Learn-p5/media/set.png)

The color can be in several formats. Above, we're using an array of 4 RGBA values, but you can also use a single greyscale value or a [p5.js color element](https://p5js.org/reference/#/p5/color).

Just calling the `set()` function doesn't change what you see&mdash;to reflect changes onto the canvas, you need to use the `updatePixels()` function after the `set()` call.

```js
let red = [255, 0, 0, 255];
set(0, 0, red); // Pixel is not actually changed on screen
updatePixels(); // Now pixel is changed!
```

If you use `set()` multiple times, calling `updatePixels()` once will reflect all those changes.

```js
let red = [255, 0, 0, 255];
set(0, 0, red); 
set(0, 1, red);
updatePixels(); // Both pixels are updated
```

The `get()` and `set()` functions also work as methods on individual image or video elements.

When using the `.set()` method on images or videos, you'll need to call it, along with the `.updatePixels()` method, before you draw them to the canvas.

```js
let red = [255, 0, 0, 255];
img.set(50, 50, red); // Pixel is not actually changed in our image element
img.updatePixels(); // Now pixel is changed in our pixel element
image(img, 0, 0); // Now our modified image element is on the screen
```

Because they're time-intensive, `get()` and `set()` are best suited for conveniently accessing a few pixels at a time. Later, we'll see how to do larger-scale manipulations, such as accessing pixels in very large images and/or doing manipulations repeatedly in the `draw()` loop.

Introduction to Pixel Manipulation

Unlike the primitive shapes we've drawn so far, media, like images, are considered external assets. To include them in our sketches, we will need to load them using special, built-in p5.js functions.

To load an image, use the `loadImage()` function. It requires one argument—the path to the image in your working directory.

```js
img = loadImage('imageName.jpg');
```

The `loadImage()` function returns a [p5.js Image](https://p5js.org/reference/#/p5.Image) element. Saving it to a variable lets us reuse the loaded image as many times as we want.

After loading an image, we use the `image()` function to draw it. The function requires three arguments: the p5.js Image element and the x and y positions where the image should be drawn on the canvas.

```js
//Draw img to the top left corner of the canvas
image(img, 0, 0);
```

Optionally, the `image()` function can take other arguments. We can also resize the drawn image to a certain width and height, as follows:

```js
let drawnWidth = 100;
let drawnHeight = 100;
image(img, 0, 0, drawnWidth, drawnHeight);
```

We can also add an additional four arguments to draw a cropped region of the image—check out the [p5.js documentation](https://p5js.org/reference/#/p5/image) to learn more.


You can also load and draw animated GIFs into the canvas using `loadImage()` and `image()`. To have them animate, you specifically need to put them inside the `draw()` loop. They're essentially multiple images packed into one, so calling `image()` in the `draw()` loop will draw the latest frame of the GIF, allowing it to animate.

```js
draw() {
  image(myAnimatedGIF, 0, 0);
}
```

Because drawing an image requires it to be completely loaded, it's important to note how and when these functions are called.

Loading images takes time. In most cases, the `loadImage()` function won't finish loading an image before the next lines of code are run—including any code that is supposed to draw the image.

Sometimes this is OK, but it can also potentially lead to unintended behavior. Later, we'll learn to work around this problem by having more control over the loading of the images.

Loading and Drawing Images

Once we have the `pixels` array, we need to know how to read and write to it. That requires correctly indexing into it to get the color values for each pixel.

Because each pixel has four color values—starting with red—we can use the following equation to find the index of the red value for a pixel at location (x,y), on a canvas that's `width` pixels wide.

```js
let indexOfRed = (x + y * width) * 4;
```

Image or video elements have their own dimensions, accessed through the `.width` and `.height` properties. Accordingly, their equations will be slightly different:

```js
let indexOfRedForImage = (x + y * img.width) * 4;
let indexOfRedForVideo = (x + y * video.width) * 4;
```

In this example, we'll access the `.pixels` array of an image. With the index of the red value, we can then get each color value, keeping in mind the R-G-B-A sequence.

```js
let red = img.pixels[indexOfRed];
let green = img.pixels[indexOfRed + 1];
let blue = img.pixels[indexOfRed + 2];
let alpha = img.pixels[indexOfRed + 3];
```

To set a new color for that pixel at (x,y), set its RGBA elements in the ``.pixels`` array to new values.

```js
img.pixels[indexOfRed] = 255; // Red
img.pixels[indexOfRed + 1] = 0; // Green
img.pixels[indexOfRed + 2] = 0; // Blue
img.pixels[indexOfRed + 3] = 255; // Alpha
img.updatePixels();
```

Above, we set the pixel to red, i.e. `[255, 0 , 0, 255]`. Again, we must call the `.updatePixels()` method (or `updatePixels()` function, if we're manipulating the canvas) to reflect changes.

Most likely, we'll want to manipulate many pixels—that's what the `pixels` array is designed for!

```js
//Double "for" loop to iterate through pixels, row by row:

let stepSize = 1;
//Loop through rows
for (let y = 0; y < img.height; y += stepSize){
  //Loop through columns
  for (let x = 0; x < img.width; x +=  stepSize){
    //Get index # of red value for pixel
    let indexOfRed = (x + y * img.width) * 4;
    //Set every pixel to black
    img.pixels[indexOfRed] = 0; // Red
    img.pixels[indexOfRed + 1] = 0; // Green
    img.pixels[indexOfRed + 2] = 0; // Blue
    img.pixels[indexOfRed + 3] = 255; // Alpha
  }
}
img.updatePixels();
```

Here, we used a double `for` loop, bounded by the image's width and height, to iterate through every pixel in the image, row-by-row. By setting its RGBA values accordingly, each pixel is set to the color black, i.e. `[0, 0, 0, 255]`. Depending on the application, you might want to go through every couple of pixels, instead of every single one. To do that, you can increase the value of the `stepSize` variable.

We called the `.updatePixels()` method outside and after this loop so that we can update all of the changes we made to `.pixels` with a single call. If we call it inside the loop, we might crash our sketch!