Streaming Microphone Input With Flask

  |   Source

This post describes how to set up a microphone stream using the HTML 5 API for handling the input, and Web Sockets for handling communication with the Flask application.

I use this for my project on chord classification with neural networks.

Capturing and sending client audio

The easiest way to capture audio is with HTML5 and javascript. For more on this, check out this tutorial on HTML 5 Rocks.

Simply call:

navigator.getUserMedia({audio: true}, initializeRecorder, errorCallback);

If successfull, this will call initializeRecorder function with the audio stream.

function initializeRecorder(strean){
   audio_context = new AudioContext;
   sampleRate = audio_context.sampleRate;
   var audioInput = audio_context.createMediaStreamSource(stream);

   console.log("Created media stream.");

   var bufferSize = 4096;
   // record only 1 channel
   var recorder = audio_context.createScriptProcessor(bufferSize, 1, 1);
   // specify the processing function
   recorder.onaudioprocess = recorderProcess;
   // connect stream to our recorder
   // connect our recorder to the previous destination

The recorder is initialized and will call recorderProcess everytime the buffer is filled with samples. Now let's set up the WebSocket, in order to communicate with the server (see also this post):

var ws = new WebSocket('ws://');

ws.onopen = function(evt) {
  console.log('Connected to websocket.');

  // First message: send the sample rate
  ws.send("sample rate:" + sampleRate);

  navigator.getUserMedia({audio: true, video: false}, initializeRecorder, function(e) {
   console.log('No live audio input: ' + e);

Depending on what you want to do with the data on the server, it might be interesting to convert it from Float32 to Int16 before sending it. That way the data will be a byte string of signed 2 byte integers.

function recorderProcess(e) {
  if (recording){
    var left = e.inputBuffer.getChannelData(0);

function convertFloat32ToInt16(buffer) {
  l = buffer.length;
  buf = new Int16Array(l);
  while (l--) {
    buf[l] = Math.min(1, buffer[l])*0x7FFF;
  return buf.buffer;

Setting up Flask

In order to listen to the websocket connection from Flask, you can use this package:

Now add the web socket listener to the Flask app:

ws = GeventWebSocket(app)

def audio(ws):
   first_message = True
   total_msg = ""
   sample_rate = 0

   while True:
      msg = ws.receive()

      if first_message and msg is not None: # the first message should be the sample rate
         sample_rate = getSampleRate(msg)
         first_message = False
      elif msg is not None:
         audio_as_int_array = numpy.frombuffer(msg, 'i2')

That's it! Every time the buffer size is reached, Flask will receive the audio.