Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 36 ยท Media Capture and Streams

Capturing from Canvas and Web Audio

Create WebRTC media without a camera: canvas.captureStream for generated video, Web Audio MediaStreamDestination for synthesized sound, and a video element's captureStream.

Cameras and microphones are not the only source of media. A canvas can produce video frames from drawing code, the Web Audio API can produce sound from oscillators or decoded files, and a <video> element can re-publish what it is playing. Each source hands you a MediaStream, which an RTCPeerConnection treats exactly like camera media.

This is useful in three situations: testing (no hardware needed), effects (draw a virtual background or overlays and send the result), and content (stream a game, a slideshow, a whiteboard or a music track).

Video from a canvas

canvas.captureStream(frameRate) returns a stream with one live video track. The track emits a frame whenever the canvas content changes (at most frameRate times per second).

JavaScript
const stream = canvas.captureStream(30);

If you pass 0, the track only emits a frame when you call track.requestFrame(). That is handy for slideshows and whiteboards, where the picture rarely changes.

A moving picture from canvas to peer to video element

The left canvas is the source; the right video element shows the frames after they were encoded as VP8 (or another codec), packetised, sent through a real WebRTC pipeline and decoded again. After about two seconds the script prints the remote size, 240x135.

Effects and overlays

Because you control the canvas, you can draw the camera into it (ctx.drawImage(videoElement, 0, 0)), add text, blur a background or draw a logo, and then send the canvas stream. That is the simplest way to build a "virtual background" or watermark. The cost is CPU: drawing and re-encoding every frame in JavaScript, so keep resolution modest.

Audio from Web Audio

The Web Audio API builds sound from graphs of nodes. A MediaStreamAudioDestinationNode is the end of a graph that exposes the audio as a MediaStream:

JavaScript
const audioContext = new AudioContext();
const destination = audioContext.createMediaStreamDestination();
oscillator.connect(destination);            // or any node
const audioTrack = destination.stream.getAudioTracks()[0];

Two practical points:

  • Browsers start an AudioContext in the suspended state until the user interacts with the page. Call audioContext.resume() inside a click handler.
  • Do not connect the destination node to audioContext.destination as well unless you want to hear it locally.
Synthesised audio track

Click the button and the output confirms the tone became a normal audio track that crossed the peer connection. (The tone is not played to your speakers, because the graph ends at the stream destination.)

Mixing audio

To send both your microphone and background music as one track, connect several sources to the same destination:

JavaScript
const micSource = audioContext.createMediaStreamSource(micStream);
micSource.connect(destination);
musicNode.connect(destination);

This mixing graph is also the right place for effects: gain control, a compressor, a filter that removes low rumble, or an analyser that drives a "who is talking" meter.

Capturing from a media element

<video> and <audio> elements have captureStream() too. It re-publishes whatever the element is currently playing as a live stream, which is how you can broadcast a pre-recorded video as if it were a camera:

JavaScript
const stream = videoElement.captureStream();     // Firefox: mozCaptureStream() in older versions

Remember that playback must be running for frames to flow, and cross-origin media without CORS headers cannot be captured. In the preview, prefer canvas and Web Audio sources.

Choosing the source in your app

A neat architecture keeps the source behind one function:

JavaScript
async function getLocalMedia(kind) {
  if (kind === 'camera') return navigator.mediaDevices.getUserMedia({ video: true, audio: true });
  if (kind === 'screen') return navigator.mediaDevices.getDisplayMedia({ video: true });
  if (kind === 'test') return makeTestStream();       // canvas + oscillator
  throw new Error('unknown source ' + kind);
}

The rest of your code only sees a MediaStream. That separation makes demos, automated tests and virtual cameras trivial to add.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does canvas.captureStream(30) return?

    captureStream returns a MediaStream whose video track emits frames as the canvas changes, up to the given rate.
  2. 2. Which node exposes a Web Audio graph as a MediaStream?

    createMediaStreamDestination() gives a node whose .stream contains the audio track.
  3. 3. Why does an AudioContext often start as suspended?

    Autoplay policies require a user gesture; call audioContext.resume() from a click handler.
  4. 4. How can you send microphone audio and music as ONE audio track?

    A Web Audio graph can mix several sources into one destination, whose stream carries the mixed track.

Summary

  • canvas.captureStream() turns drawing code into a video track; frame rate 0 plus requestFrame() suits slideshows.
  • AudioContext.createMediaStreamDestination() turns a Web Audio graph into an audio track, and can mix several sources.
  • Media elements can also be captured with captureStream().
  • Hide the source behind one function so camera, screen and test sources are interchangeable.

Next: Screen Capture and Recording