Measuring the heart beat using a smart phone

Long time, no blog posts. Four days short of two years, actually. It almost feels bad to ruin the imaginary anniversary party but luckily it's not illegal to have cake anyway.

So, today I was typing a IM and decided to deliver a mediocre joke in form of a hasty sketch. For the canvas I though I could take black photo by covering the camera of my phone with my palm. However, it turned out to be more difficult than I thought since the camera adjusted its light sensitivity so much that it picked up the light scattering from the inside of the palm making the picture always reddish in color. 

This gave me an idea to test whether I can measure my heart rate by recording video from my palm. I put on the flashlight on my trusty Honor 6 and recorded a short video with the camera pressed against my palm. What I got was hundreds of frames looking like this:

Representative example frame of the raw video data

Although one doesn't see much just looking at the video, a more careful analysis with Python reveals that there is in fact some interesting time behaviour hidden under all that redness. By taking the mean value of the red channel (with 8-bit value ranging from 0-255) over all 1920x1080 pixels, the following signal is obtained:

So, the peak-to-peak variation is small, about 1.5 per pixel (or 0.6% compared to the background 'redness') but it is there and it is very clear. That's the variation of scattered intensity of red light due to varying amount of blood in veins during the pump cycle of the heart. From that we could obtain the rate by calculating the number of beats per time interval but let's use a more sophisticated method instead.

Mathematically every signal can be constructed by summing up a suitable set of sine waves. To find the multipliers and the phases of the waves, we use something called the Fourier transform. The Internet is full of excellent explanations what it is and how it works, for example this Youtube video by 3Blue1Brown, so I'm not going to go it through here. The important part here is to know that taking the absolute value of the Fourier transform and squaring it gives tells us how the power of the signal is distributed between the different frequency components of the signal.

To get of rid of the constant background which would cause a huge zero frequency component to the power spectrum, the mean value is subtracted from the red channel signal before taking the Fourier transform. The resulting power spectrum looks like this:

The first larger peak is now the one we are interested in, for it is the base frequency of the signal and thus corresponds to the heart rate. The peak value is close to 85 bpm (beats per minute) which actually matches very well the value measured by my (in)activity bracelet. Not bad, eh?

We also see another peak at 170 bpm which is twice the base frequency. The integer multiples of the base frequency, called harmonics, usually appear in the power spectrum of periodic signals, unless they are pure sine waves. However, interesting fact to me is that there is very little contribution to the signal from higher harmonics. The signal is thus mainly composed of the base frequency and its first harmonic with twice the base frequency. To me this makes sense since the blood pressure is mainly affected by ventricular chambers which either contract (systole) or relax (diastole). However, don't take this too seriously since I'm not a physician but a physicist, so there's a somewhat realistic chance that I might be oversimplifying the cardiovascular system a bit too much..

After a long break it was fun to write something to the blog again. I hope that the next update won't take quite as long. Lenk's quest is not dead either, and has even progressed a bit since the last time. If the stars align right, we might see the next part soon..

Below you find the Python code I used in the analysis of the video data:

from __future__ import division, print_function

import numpy as np
import matplotlib.pyplot as plt
from moviepy.editor import VideoFileClip

clip = VideoFileClip('source.mp4',audio=False)
fps = clip.fps

data = []

#Compute the mean value of red channel per frame
for frame in clip.iter_frames():
    data.append(np.mean(np.mean(frame[:,:,0],axis=1)))

data = np.array(data)

data = data[700:826]
time = np.arange(data.size)/fps #time relative to frame number 700

#Calculate the Fourier power spectrum
#The mean value is substracted in order to remove the zero frequency component
fourier = np.abs(np.fft.fft(data - np.mean(data)))#**2
freq = np.fft.fftfreq(data.size,1/fps)

plt.plot(time,data)
plt.xlabel("Time (s)")
plt.ylabel("Mean red channel signal")

plt.figure()
plt.axvline(x=85,label='Heart rate sensor (85 bpm)',color='red')
plt.axvline(x=2*85,label='2x Heart rate sensor (170 bpm)',color='orange')
plt.axvline(x=3*85,label='3x Heart rate sensor (255 bpm)',color='green')
plt.plot(freq[:freq.size//2]*60,fourier[:freq.size//2],label="Fourier transform",color='blue',linewidth=2)
plt.legend()

plt.xlabel("Frequency (1/min)")
plt.ylabel("Fourier power spectrum |FFT|$^2$")

plt.show()

Lenk's Quest part II: Controlling the display with AVR

Earlier I managed to get the LCD1602 display with I2C module working with Raspberry Pi and Python (see part I and part II). Now the challenge is to repeat the same thing with ATtiny and C and take it a step further.

As a disclaimer, if you are interested to make a program on AVR using a display and following this (or any other) post as a tutorial, I have one advice: don't do it blindly. I think it is quite obvious that I'm not a pro at this (especially after this post) and while my technical posts are meant to work as guidelines, they are surely not tutorials. With that in mind, read critically and don't repeat my mistakes.

AVR has this so-called Universal Serial Interface (USI) which allows the serial communication on the hardware level. Compared to software implementations, the USI allows faster communication requiring less code. The USI has two modes; three- and two-wire mode. The two-wire mode (TWI) is identical with I2C but apparently owing to the copyright issues or something it can't be called as such.

My first plan was thus to use TWI instead of writing my own code. It's built-in so it's probably easy to use, right? Soon after I took out the datasheet, I came into a conclusion that I don't want to stress out my brain cells to figure that shit out. I'll probably regret my decision later but it was Sunday then after all.

I switched to the plan B that was to get out my Python code and translate it to C. Implementing a serial protocol in software instead of hardware is called bit banging by the way. Totally unrelated, it brings to my mind a certain Finnish word, 'pilkunnussija', that is used for a person who is unnecessarily keen to detail. Literally translating to 'comma fucker', it is somewhat synonymous to 'grammar Nazi', except with an extra serving of heat and lubrication.

Up to this point, I've been writing all my code into a single .c file but it is not a very good practice for a larger project like Lenk's Quest. To make the source code more manageable, I'll split it into multiple files. There are two parts in controlling the screen: I2C communication and the display specific commands. Thus it is natural make a separate file for I2C related functions and definitions that are used by the screen control functions in another file.

Before going through the source code, I want make a small detour. If one desires to use multiple .c files in their program, they have accompany them with header files as well. Header files (.h) contain macro definitions (i.e shorthands and mnemonics) and function declarations. But why are they needed?

Suppose you use the function eat_apple in your code that is defined in another file. Without including the appropriate header file, there would be no way for the compiler know what the function takes in and what it outputs. In C this kind of ignorance is not tolerated. The header file with the line double eat_apple(int); removes this problem. The .c file actually containing the definition of the function eat_apple is not needed at this point; in the extreme case it doesn't even have to exist yet! It is not required until it needs to be compiled to be linked together with all the other source code files, forming a single executable program. (More info about the headers and linking, see here and here.)

I2C Communication

As with my Python code, I need only unidirectional I2C communication: from the microcontroller to the screen. I want to keep it simple, but as in the future I might want recycle the code, I want it make it general purpose as well. (Always when programming, it is a good practice to try to keep the code general rather than specific. It makes reusing and modifying the code easier and helps to develop your thinking process.)

My header file i2c.h looks like this:

#ifndef I2C_H_
#define I2C_H_

#include <avr/io.h>

#include <util/delay.h>

#define SDA_PIN 4
#define SCL_PIN 3

void init_i2c();
void write_bytes(uint8_t address, uint8_t *bytes);

#endif

The first two lines together with the very last one form the header guard that prevents the file to be included multiple times (which would almost certainly happen when dealing with more than one file). Inside the guard I first include some AVR libraries that'll I need later on in the source code file i2c.c.

The next two lines define macros for the pins I'll be using for the communication. This sets PB4 as the data line (SDA) and PB3 as the clock line (SCL). Owing to the includes, I could have written this alternatively like this:

#define SDA_PIN PB4
#define SCL_PIN PB3

Then come the function declarations. The first one is the initialisation function init_i2c that sets the defined SDA_PIN and SCL_PIN as outputs and their states to HIGH (I2C bus ready state). The second one, write_bytes, sends a sequence of bytes to the given address. bytes is a pointer that is interpreted to be a null-terminated string: something I thought was a smart choice because that way I don't have to count the number of bytes and pass it to the function separately. It eventually turned out be, erm, not so smart after all but I'll get back to it later.

Let's move on to the actual source code in i2c.c. The beginning of the file looks like this:

#include "i2c.h"

void init_i2c(){

    //set SDA and SCL as output
    DDRB |= (1 << SDA_PIN)|(1 << SCL_PIN);

    //set both high
    PORTB |= (1 << SDA_PIN)|(1 << SCL_PIN);
    _delay_us(10);

}

Nothing unusual here. SDA_PIN and SCL_PIN dictate the positions of the pins and the bits in DDRB and PORTB are set accordingly.

As MCU is much faster toggling its pins than Raspberry Pi with Python, I've added some delays to make sure that the receiving party can keep up with the speed. They shouldn't be longer than necessary, as it slows down the communication speed. The ones you see here are surely not optimised; I just picked up a safe value and showed them up everywhere I could as I wanted avoid any bugs owing to too fast pin toggling. There are probably some kind of guidelines which I'll look up later on (or not, depending whether the current solution is fast enough. There's no need to optimize if the solution already meets the requirements).

The write_bytes function uses an auxiliary function called write_byte which writes a single byte to the I2C bus and reads and returns the acknowledge bit:

int write_byte(uint8_t byte){
    //write a single byte to the bus

    //clock down
    PORTB &= ~(1 << SCL_PIN);
    _delay_us(10);

    int i;
    for(i=7;i>-1;i--){
        //set SDA
        if(byte & (1 << i))
            PORTB |= (1 << SDA_PIN);
        else
            PORTB &= ~(1 << SDA_PIN);
        _delay_us(10);

        //clock high
        PORTB |= (1 << SCL_PIN);
        _delay_us(10);

        //clock down
        PORTB &= ~(1 << SCL_PIN);
        _delay_us(10);

    }
    //read ACK

    //set SDA as input
    DDRB &= ~(1 << SDA_PIN);
    PORTB |= (1 << SDA_PIN);
    _delay_us(10);

    //get ACK

    //clock high
    PORTB |= (1 << SCL_PIN);
    _delay_us(10);

    int ack = PINB & (1 << SDA_PIN);
    _delay_us(10);

    //clock low
    PORTB &= ~(1 << SCL_PIN);
    _delay_us(10);

    //set SDA as output
    DDRB |= (1 << SDA_PIN);
    PORTB |= (1 << SDA_PIN);
    _delay_us(10);
    
    return ack;
}

This is very similar to the insides of the loop in the write function I wrote in Python. The main difference is that instead of the list of ones and zeros, a byte is represented by an uint8_t variable. The write_bytes function utilising this is:

void write_bytes(uint8_t address, uint8_t *bytes){
    //Start condition (bus should be ready)
    PORTB &= ~(1 << SDA_PIN);
    _delay_us(10);

    DDRB |= 1;
    //write the address + read bit
    write_byte(address<<1);

    //write bytes
    int i;
    for(i=0; bytes[i] != '\0' ; i++)
        write_byte(bytes[i]);

    //Stop condition    
    _delay_us(10);

    PORTB &= ~(1 << SDA_PIN);
    _delay_us(10);
    PORTB |= (1 << SCL_PIN);
    _delay_us(10);
    PORTB |= (1 << SDA_PIN);
    _delay_us(10);

}

Here, after the start condition, the loop goes through the bytes in the array until the null terminator is encountered and the transmission is terminated with the stop condition.

Before I started to make the display control code upon write_bytes, I tested it out with a short code that turns the backlight off and on again, like with the Python code. Not surprisingly, the screen didn't respond at all.

To pinpoint the porblem [sic], I first wanted to know whether the I2C chip responds correctly to the sent data. Easiest way to do this is to read the acknoledge bit but, unlike with the Pi, I had no screen to print its state to. Thus I added a debug LED to my circuit that lit up according to the value returned by write_byte. It revealed that the signal wasn't getting through which was caused by an programming error in the clock signal (one HIGH state was accidentially LOW which is effectively a missing pulse).

But the screen still didn't turn off. This one was harder to spot because it wasn't just a random error but a design flaw. I tried setting the screen of by sending a zero byte to the I2C chip. Guess what 0 stands for as well? The null terminator. So my initially smart solution turned out to be crap as it doesn't allow sending a zero byte over I2C at all which is obviously a serious shortcoming. It doesn't prevent any of the display's features though, so go on with this and fix it sometime later.

Display control

As we found out earlier, the display has to be used in a 4-bit mode with the I2C chip. Commands are sent in two parts, each sent three times toggling the enable bit. To simplify the use of the display, we'll write a bunch of helpful functions to control it.

The header file display.h is written in the same manner as before:

#ifndef DISPLAY_H_
#define DISPLAY_H_

#define LCD_ADDRESS 0b0100111

#include "i2c.h"

void init_display();
void write_instruction(uint8_t rs, uint8_t rw, uint8_t cmd, uint8_t backlight);
void write_text(uint8_t *str);
void write_text_2line(uint8_t *str,uint8_t *str2);

#endif

The file again starts and ends with the guard. Then I define a macro for the address of display's I2C module and include the I2C code freshly out of the oven.

After the power-on, the screen has to be initialised according to the procedure described in its datasheet. With my code this is done followingly:

#include "display.h"

void init_display(){

    //init i2c bus
    init_i2c();

    //50 ms delay (required) 
    _delay_ms(50);

    //initcommand
    write_bytes(LCD_ADDRESS, "\x38\x3c\x38");
   
    //5 ms delay (required) 
    _delay_ms(5);
  
    //Init command repeated twice more according to datasheet 
    write_bytes(LCD_ADDRESS, "\x38\x3c\x38");
    write_bytes(LCD_ADDRESS, "\x38\x3c\x38");
   
    //Set 4-bit interface  
    write_bytes(LCD_ADDRESS, "\x28\x2c\x28");

}

The string literal "\x38\x3c\x38" is same as the array {0x38,0x3C,0x38,'\0'}. The function write_bytes addresses the device marked by LCD_ADDRESS, sends these bytes (0x38 with the enable bit LOW and 0x3C HIGH) to the I2C bus and stops on the null terminator ('\0' = 0x00). After setting the display to the 4-bit mode, the initialisation is complete and we can start sending instructions:

void write_instruction(uint8_t rs, uint8_t rw, uint8_t cmd, uint8_t backlight){
    uint8_t hbyte, lbyte, rwrs;

    rwrs = 0;
    if(backlight)
        rwrs |= 0x08;
    if(rs)
        rwrs |= 0x01;
    if(rw)
        rwrs |= 0x02;
    

    hbyte =  (0xf0 & cmd)|rwrs;
    lbyte =  (0xf0 & (cmd<<4))|rwrs;

    uint8_t byte_str[7] = {hbyte,hbyte|0x04,hbyte,lbyte,lbyte|0x04,lbyte,'\0'};

    write_bytes(LCD_ADDRESS, byte_str);

}

The input of the function is formatted so that it follows the format of the commands given in the datasheet. Thus rs and rw are merely truth values, whereas cmd is the data byte that is split into two 4-bit parts.

And finally we've reach the the point we've been seeking for: writing text! In the Python test program I sent all the characters as arrays of bits which was very cumbersome. Since the charcter table of the display is compatible with ASCII codes, we can simply make a plain text string and send character by character. Non-ASCII and custom characters can be used with \x as earlier.

void write_text(uint8_t *str){

    int i;
    for(i=0;str[i] != '\0';i++)
        write_instruction(1, 0, str[i], 1);

}

void write_text_2line(uint8_t *str,uint8_t *str2){
    //DDRAM address 1st line
    write_instruction(0, 0, 0x80, 1);
    write_text(str);
    //DDRAM address 2nd line
    write_instruction(0, 0, 0xC0, 1);
    write_text(str2);

}

The first function is a bit stupid. It just starts writing wherever the current address is pointing at. The second one, on the other hand, first sets the Display Data RAM (DDRAM) address to the beginnings of the rows before writing.

Test Program

Here's a small program to test it out:

#define F_CPU 1000000UL 

#include <util/delay.h>

#include "display.h"


void main(){

    init_display();

    write_instruction(0, 0, 0x28, 1); //Function set (4-bit, 2-line, 5x8 dots)  
    write_instruction(0, 0, 0x08, 1); //Display Off, cursor off, blinking off 
    write_instruction(0, 0, 0x01, 1); //Clear display  
    write_instruction(0, 0, 0x06, 1); //Entry mode (left to right, don't shift display)

  
    write_instruction(0,0,0x0C,1); //Display on     

    write_text_2line("<LENK'S QUEST|=O","Graphics test   ");
 
    while(1){};
    
}

The program starts naturally with the display's initialisation function. It is followed by a bunch of settings after which it is set on and ready to be written. And here's the result:

Despite all the deficiencies of the code, it works like a charm! But to be honest, that ASCII sword surrounding the title looks a bit crude. I think graphics need a bit of seasoning..

Lenk's Quest part I: Game design

Not all those who wander are lost 

J. R. R. Tolkien

A good adventure doesn't require a map but at times one can be quite handy. Especially in the case where one is building something complicated out of components that are not. In order decrease the likelihood of finding Lenk's Quest deep in a swamp populated by spirits of never finished projects, I present thee the Document of Design.

Project's roadmap, artist's interpretation

Hardware limitations

Programming microcontrollers differs somewhat from programming a personal computer. With computers the question I repeatedly ask myself is how can I do this? Whatever I want to do they have the juice for it.

Microcontrollers, on the other hand, are not steam engines that can push through everything. They are more like low-profile Segways that get stuck on the first pebble they come across. More appropriate question is what can be done? The programmer needs to turn their brain into the Haiku-mode and try to come up with creative solutions to squeeze the most out of the scarce resources and stringent limitations. I find this a refreshing challenge.

So, what do we have? First of all, I want to use my 16x2 character display (LCD1602) which needs two pins for I2C communication. If I had similar I2C I/O expander chips like the display has, I could connect as many buttons as I wanted (8 per chip) and still use the same two wires as the display does. But I don't, neither do I have any other sensible mean of serial communication.

Thus using ATtiny85 I have 3 free I/O pins for buttons, one per each. Not too bad. I only get half a D-pad but considering that my screen has only two rows of text, I think I can cope with that.

What about sound effects and music? I have a few ideas but I haven't tested them yet. There are some technical complications that I'll go through in a separate post. Getting an earworm infection from a suitable tune has suffice for this project.

Program flow

The program structure is similar as in The Torment of Alfred McSilvernuts. In the main program I'll first initialise everything, like configuring the pin directions and setting up the counter. Then the execution moves on to the infinite loop that does nothing.

The loop is interrupted by a timer that overflows approximately every 16.7 ms, aiming for the frame rate of 60 per second. All the action happens in the Interrupt Service Routine where the chip reads whether the buttons are pressed or not, executes the next of the game logic, and draws stuff on the display.

To make the game more interesting, I want to have more than just a one level. I also want that my game has a title screen that pops up at the start-up, and an ending screen that tells the player whether they have won or lost the game. I think the best way to do this is to break the game logic section into separate pieces for different states, like this:

What comes to computer games, I believe it's a common practice to separate the graphics from the game logic. However, with my hardware updating the whole screen takes a relatively long time that makes the display flicker. Not a good thing for the game play experience.

Fortunately I don't have to update the whole display every frame as normally only small portions of the screen change. I wouldn't want to mix the graphics into the logic but, for now, it seems to be the smartest solution to update the part of the screen immediately after the corresponding part of the game logic (e.g. character moves a step) is computed. I'm not completely happy with it but this way I don't have to think how to store and pass the information to the separate update function.

Game mechanics, objective and dungeon design

As a Legend of Zelda ripoff inspired piece, the 'camera' will be viewing the game area from the bird's perspective. Since I have only 3 buttons and two rows of text on the screen, it is best to limit the rooms (or levels) to a single dimension: the horizontal one. This way I need only two buttons, left and right, for the moving. The third button can be then used for different kind of actions, like attacking with the sword. I'll call it, not very surprisingly, the action button or the A button for short.

As the movement is limited on the one line, the other one on the screen can be used to present the wall of the room. There are some positions on the room where pressing A will make the player switch the room instead of attacking. These spots are indicated to the player by placing a door on the wall right next to them.

To make the game a game, we need to add some challenge. Let's say that the objective of the player is to escape the dungeon. This can be done through a locked door. The player has to get the key but it is guarded by a ferocious werewolf who tries the kill the player. To get the key, the player needs to slay it first.

Putting all these ideas on the paper, the dungeon turned out to look like this:

I decided to settle for two rooms. This way the middle row (one with the open door) is visible all the time: on the top row of the screen when in the lower room and on the bottom row when in the upper room. This gives the player a sense of the shape of the dungeon. If the wall was always on the same row, the player would have hard time to get a grasp of the geometry as the camera would appear to turn 180 degrees around every time they pass through a door. This solution is not limited only to two rooms. I could add an arbitrary number of rooms like this:

I just don't have any real content to be put into them. And nothing's more frustrating in video games than doing something totally worthless. It's like taking The Hobbit and stretch it into three movies.

The rooms are short so they fit into a single screen. This way I don't have to implement camera scrolling. It would not be very complicated, but certainly more than my game needs.

You can see that the door is positioned in the middle of the screen, two tiles away from the player's start position. This way the player can test and get used to controls before entering the second room. There's also a suitable distance between the door and the werewolf, so the player won't get their ass kicked immediately at the doorstep.

The last thing I want to mention about the dungeon design, is the bottom left corner of the screen. You may have noticed that the lower room is a bit shorter than the upper one. That's because I need a place for the health counter. It is nice to have all the counters at the same position on the screen at all times, so I made a part of the level impassable, so the player can't step on the counter.

---

Now, with the roadmap of objectives established, we're ready get our hands dirty. I think the best way to continue is to get started with graphics. After all, the easiest way to make sure that the things are working is to see them in action.

Obscure systems and a Zelda rip-off

New Legend of Zelda is coming out next year. As a long term fan, I'm so excited that I already went and bought myself a Wii U. It is my actually my first console since I moved out on my own. As a matter of fact, I've been more interested in making games than playing them in the recent years. But Breath of the Wild is something I don't want to miss. Look at all this vast, natural world, puzzles utilizing the physics engine, and ancient technology!

Yeah, I'm drooling (figuratively). I don't know whether I should have waited for the new console, Nintendo NX, but considering that nobody does really have a clue about it, I don't think my purchase was a bad one.

Wii U has a bunch of nice looking titles, like extremely well received Super Mario 3D World of which some say it is the best Mario game ever. It is also backward compatible with Wii and has Virtual console for the old classics. Apparently it also runs Gamecube games after a bit of tweaking but doing so puts you on Santa's Naughty list...right? Probably voids the warranty as well.

On top of that Wii U has sold rather badly, so in a few years it is going to be really hard to find. Every now and then I get excited about some specific retro system. For example, 8 years ago I read about Game Boy Micro and I was like Oo, a smaller backlit version of GBA! If I get an SD card reader for that, it would be so handy to carry around and play all the best Game Boy games whenever I feel like it (smartphones weren't so smart back then). I've got to have one!

Blue Game Boy Micro and GBA cartridge [source]

Never heard of it? No wonder, as it sold only 2.42 million units worldwide. For comparison Game Boy Advance has sold 35.52 million units and this doesn't include SP. I failed to find it from any game stores, first or second hand, so I eventually bought it off eBay. I personally like it more than Advance or SP. It is smaller but the screen is better looking and it feels good in hand. It doesn't support Game Boy and Game Boy Color games (unless one uses an SD card reader, naughty naughty), which might have contributed to its failure together with the release of DS. It's a shame but this time I'm prepared if the same thing were to happen to Wii U.

But even if they'd outlaw Wii U, Breath of the Wild won't be out until the next year. I don't think I can wait that long. That's why I'm going to put my recently acquired electronics skillz to use and make the next Zelda adventure myself...except I don't dare to call it a 'Zelda'. There are rather high expectations of the fans that my little project will never meet. But even more importantly, Nintendo happens to be very protective of their electronic rights and don't take violations kindly (I had to allow ads on my last video if I wanted to use a tune from Super Mario Bros 2).

Following the great success of The Torment of Alfred McSilvernuts, I hereby announce my next adventure game:

It would surely look like this if I had more pixels and, like, colors.

(Although it might appear like one, it is totally not a fart joke here. I'm a Master of Science after all; my sense of humor is delicate and mature as fuck.)

Unlike the last game with a bunch of LEDs lighting up, this one requires a bit more thought. Just making it is going to be an adventure on its own. I hope the difficulty setting is no higher than medium.

Snakes and displays part II: Writing text on the screen

Last time we learned how to talk with the I/O expander chip via I2C protocol. Now we'll see whether we can figure out how to get some text on the screen.

First we have to know how the data bits sent to the I2C chip are mapped to the pins of the LCD controller. Let's take another look at the schematic:

So, when a data byte is sent to the expander, it sets the pins P7...0 accordingly. Apparently the high bits 7...4 are mapped one-to-one to the 4 highest bits of the display's data bus. Bit 3 is controls the backlight of the screen through an NPN transistor as we concluded in the previous post. 

Bit 2 is connected to CS which is supposed to stand for Chip Select, I guess. In the datasheet the pin is called E for Enable and is used to indicate the chip that we want to read/write data to it. Bit 1 controls the RW (Read/Write) pin, indicating the direction of data, and bit 0 is RS (Register Select) which tells whether we want to access the settings of the display or the data (text) memory.

Okay but what about the low half DB3...0? According to the datasheet there are two interface modes. In the 8-bit mode the instructions and data are transferred using all the DB pins. However, it also has the 4-bit mode where the byte going through DB is split into two 4-bit parts that are consecutively pushed through DB7...4. So, an instruction consisting of RS, RW, and DB7...0 bit is converted to two bytes and sent through the I2C like this:

Pulling this all together, a hasty (or stupid, can't decide which one I am) person might think that one can control the screen by setting the E pin HIGH and sending the bytes over the I2C. Well, it turned out you can't.

I partly blame the schematic where the pin was misleadingly labelled as CS. To my best knowledge, the Chip Select pin usually works so that it makes the chip to listen incoming signals. Thus I thought it is simply enough to set it HIGH. If one thinks about this a bit further before implementing it (and wasting a couple of hours), they would have figured out why it can't work.

If my first though was the case, the chip couldn't differentiate between the proper input signal and, say, the transition between the aforementioned two bytes. There has to be a way to tell the chip that "The input is now settled. Go get it, boy!". Like with a clock pulse or something like that. The proper use of E pin is actually described in the datasheet, if I were to read it more carefully:

A plateful of timing spaghetti, s'il vous plâit!

So, in order to write on the LCD controller, I first have to set the RS and RW bits, then wait awhile, and send a pulse on the E pin that on the descending edge records the state of the data bus. Then using the write_bytes function from the previous post, the Python code to send an instruction looks like this:

 def write_instruction(cmd):  
     db74 = [cmd[2],cmd[3],cmd[4],cmd[5]]  
     db30 = [cmd[6],cmd[7],cmd[8],cmd[9]]  
     rwrs = [cmd[1],cmd[0]]  
   
     write_bytes([db74+[1,0]+rwrs, db74+[1,1]+rwrs, db74+[1,0]+rwrs])  
     write_bytes([db30+[1,0]+rwrs, db30+[1,1]+rwrs, db30+[1,0]+rwrs])

Both parts of the command are sent three times to the chip toggling the E bit. I didn't add any additional delays to the code, since controlling the GPIO with RPi.GPIO is rather slow (according to this the max frequency is 70 kHz = 14 microseconds per pulse) and the datasheet says the waiting times should be order of 10 nanoseconds.

However, the write_instruction function can't be used immediately after the power up, as the display will be in the 8-bit mode. The 4-bit mode has to be set with a separate command that in principle is 8-bit but it doesn't care the state of the unconnected pins. In addition, the datasheet describes a boot up sequence that has to be executed before the display can listen to other instructions. The function to initialize the display is

 def init_display():  
 
     #Init command  
     write_bytes([[0,0,1,1,1,0,0,0], [0,0,1,1,1,1,0,0], [0,0,1,1,1,0,0,0]])  
   
     #5 ms delay (required) 
     time.sleep(0.005)  
  
     #Init command repeated twice more according to datasheet 
     write_bytes([[0,0,1,1,1,0,0,0], [0,0,1,1,1,1,0,0], [0,0,1,1,1,0,0,0]])  
     write_bytes([[0,0,1,1,1,0,0,0], [0,0,1,1,1,1,0,0], [0,0,1,1,1,0,0,0]])  
   
     #Set 4-bit interface  
     write_bytes([[0,0,1,0,1,0,0,0], [0,0,1,0,1,1,0,0], [0,0,1,0,1,0,0,0]])  
   

With these we are finally all set! Let's test it out by writing some text on the screen. Sending an instruction 10XXXXXXXX writes a single character on the screen and increments/decrements the cursor position by 1. X:s stand for the 8-bit memory address of the character that are tabulated in the datasheet. For example A is 0100 0001 and & is 0010 0110 (these actually follow the ASCII coding, which will be useful later on). It is also possible to define own character graphics but I haven't yet tried it out. Here's my test code:

 #Init  
 init_display()  
 
 #Settings (part of the boot up sequence in the datasheet) 
 write_instruction([0,0,0,0,1,0,0,1,0,0]) #Function set (4-bit, 1-line, 5x10 dots)  
 write_instruction([0,0,0,0,0,0,1,0,0,0]) #Display Off, cursor off, blinking of cursor off  
 write_instruction([0,0,0,0,0,0,0,0,0,1]) #Clear display  
 write_instruction([0,0,0,0,0,0,0,1,1,0]) #Entry mode (left to right, don't shift display)  
   
 #set on display  
 write_instruction([0,0,0,0,0,0,1,1,0,0]) #Display on  
 
 #write text on the screen  
 write_instruction([1,0,0,0,1,1,1,0,0,0]) #8  
 write_instruction([1,0,0,0,1,0,1,1,0,1]) #-  
 write_instruction([1,0,0,1,0,0,0,0,1,0]) #B  
 write_instruction([1,0,0,1,0,0,1,0,0,1]) #I  
 write_instruction([1,0,0,1,0,1,0,1,0,0]) #T  
 write_instruction([1,0,0,0,1,0,0,0,0,0]) #  
 write_instruction([1,0,0,1,0,0,0,0,1,1]) #C  
 write_instruction([1,0,0,1,0,0,1,1,1,1]) #O  
 write_instruction([1,0,0,1,0,0,1,1,1,1]) #O  
 write_instruction([1,0,0,1,0,0,1,0,1,1]) #K  
 write_instruction([1,0,0,1,0,0,0,0,1,0]) #B  
 write_instruction([1,0,0,1,0,0,1,1,1,1]) #O  
 write_instruction([1,0,0,1,0,0,1,1,1,1]) #O  
 write_instruction([1,0,0,1,0,0,1,0,1,1]) #K  
   
 #Clear pins  
 GPIO.cleanup()  
   

and the magnificently unsurprising result:

Splendid. I also tried using the screen in the two-line mode but the contrast went really poor and turning the contrast screw didn't do a thing. I guess the screen requires more power in the two-line mode than the GPIO can provide. At least not through the 3.3V pin. With 5V volts the contrast was way better but then the I2C communication didn't work anymore (Rasp's IO pins are 3.3V in the HIGH state). I found out later that it is not necessarily a good idea to put 5V to GPIO pins anyway, so I advice not to mix the voltages and risk getting the smoke out of the GPIO bus.

References:

LCD 1602 Datasheet

Snakes and displays part I: Communication with I2C

Technology grows old so fast nowadays so it's really hard to keep up. My first "game console" was outdated even before it got out for consumers to enjoy! Maybe my timing was just poor. I couldn't get my system ready to be presented in this year's E3 Expo letting all those small garage companies like Nintendo, Sony, and Microsoft steal my market share.

For the next model I surely need to spice up specs too. What is the funniest thing one can make with three LEDs? Traffic lights probably, which gives away the biggest shortcoming of my console. But what would be the best choice for graphics? 3D's popular but I believe it is nothing more but a passing fad. Like who ever talks about how they remember particulary nice polygons or shaders from their favorite games? No one.

No, the best games are all about good gameplay, fascinating storylines or memorable characters like Super Mario, Link and Sonic. And what style of graphics has more characters than ASCII graphics? That's why I ordered this 16 x 2 character display LCD 1602 from DealExtreme for less than 5 euros.

Front side of the LCD with power and data wires attached.

The display consists of two clearly separable parts: the screen and its controlling electronics, and the I2C I/O expander circuit. The last one is quite handy, as it allows to move data from to the microcontroller to the display using only two wires instead of a half fuckton of pins you can see right below the screen. With 5 pin I/O of ATtiny85 (not counting the RESET pin), that's more than welcome.

Back side of the circuit. The I2C expander is the black circuit.

Before I can use this for anything, I must first learn how to use it. The manufacturer provides the C++ library to be used with Arduino development board. I read somewhere that it might work somehow with avr-gcc as well but I decided not to use it anyway. It will be more educative to figure out from scratch how to control the display.

To ease out my task, I'll begin programming with Raspberry Pi and Python instead of AVR and C. Compared to C language, Python is much more flexible and faster to write, making it ideal for prototyping. It is an interpreted language, meaning that it doesn't require compiling but on the downside isn't generally as fast to execute. However, there are powerful and efficient libraries for pretty much everything that makes Python useful for computationally demanding applications as well. At work I use Python for measurement control, data analysis and simulations, for example. And the best part is that it is free and also very easy to learn (you can find Python tutorials for example at Codecademy).

I decided to use RPi.GPIO module to control GPIO pins. On the display, there are four pins to be connected to the Pi. I connected VCC to pin 1, GND to pin 6 and the I2C lines SCL and SDA to pins 3 and 5 respectively. There are two different numbering schemes for the Pi's GPIO pins that RPi.GPIO supports. I decided to use the physical numbering i.e. how the pins are arranged on the board, as I find it easier to use.

Physical numbering of Raspberry Pi GPIO pins [source]

The connector on the screen was a bit bent during the delivery but the screen seemed to work ok as the backlight turned on when the screen was connected.

The display lights up immediately when connected to the Pi. 

Next I'd like to write some text on the display which took me two nights to figure out how. The first night was spend on understanding how to communicate through I2C protocol. I2C or IIC (Inter-Integrated Circuit) is a two-wire serial communication protocol invented by Philips Semiconductors. There are many sources out there where I2C is described in detail (e.g. this nice tutorial), so I'll just cover here the parts I need.

In the idle state both SDA (serial data) and SCL (serial clock) lines are in HIGH state. The communication starts when the master device (Raspberry Pi) sets the SDA line down while SCL is kept HIGH. This tells the slave (I2C chip of the display) to get ready to receive data. Normally SDA shouldn't change when SCL is HIGH.

As I2C supports multiple devices on the same bus, we need a way to tell which device we want to communicate with. Therefore the first byte send after the start condition contains a 7-bit address that matches the correct device. The 8th bit is a read/write bit indicating whether the master wants to send to or receive data from the slave (HIGH for read, LOW for write). The master sets the state of the SDA line according to each bit and sends the clock pulse on SCL when the bit is ready to be read.

After the 8th bit, the master gives the control of the SDA line to the slave. If there is a slave device with the correct address, it sets the SDA line LOW, indicating that it is ready to receive or send data. If not, then SDA should stay HIGH and the master decides how to continue. This as called as the acknowledge bit (ACK). The master reads the ACK bit and sends the 9th clock pulse on SCL.
If the ACK was LOW then it is time to read or write some data. The procedure for writing is pretty much the same as with the previous step; the master sets the SDA line and informs the slave with SCL pulses. After 8 bits the control of SDA is given to the slave and the ACK bit is read. Reading the data is the same, except now the slave controls the SDA line and the master sets the ACK bit. In both cases SCL is controlled by the master, however.

This can be repeated for arbitrary number of bytes. When the last byte is written/read, the master sets the SDA line LOW. Then it sets the SCL to the HIGH state and sets the SDA line HIGH as well. Change of SDA from LOW to HIGH when SCL is HIGH tells the slave that the communication is over.

And here is how I implemented it in Python:

 import RPi.GPIO as GPIO  
   
 #Pin numbering mode  
 GPIO.setmode(GPIO.BOARD)  
   
 #pins  
 SCL = 3  
 SDA = 5  
   
 #set pins to output  
 GPIO.setup(SCL,GPIO.OUT)  
 GPIO.setup(SDA,GPIO.OUT)  
   
 #set the bus to ready state  
 GPIO.output(SDA,True)  
 GPIO.output(SCL,True)  
   
 #chip address  
 ADDRESS = [0,1,0,0,1,1,1]  
   
 def write_bytes(array_of_bytes):  
  #add read/write bit to address and add the result to array   
  array_of_bytes = [ADDRESS+[0]] + array_of_bytes  
   
  #start condition (bus should be ready)  
  GPIO.output(SDA,False)   
   
  #write bytes  
  GPIO.output(SCL,False) #Clock LOW
   
  for byte in array_of_bytes:  
   for bit in byte:  
    GPIO.output(SDA,bit)  #Set data line     
    GPIO.output(SCL,True) #Clock HIGH  
    GPIO.output(SCL,False) #Clock LOW  
      
   if not byte is array_of_bytes[-1]:   
    #if the byte is not the last one -> acknowledge  
   
    GPIO.setup(SDA,GPIO.IN) #Give SDA contol to slave     
    GPIO.output(SCL,True)  #Clock HIGH
   
    print 'ACK: ', GPIO.input(SDA) #read the ACK bit   
   
    GPIO.output(SCL,False)  #Clock LOW  
   
    GPIO.setup(SDA,GPIO.OUT) #Regain the control of SDA  
    GPIO.output(SDA,True)  
   
   else:  
    #stop condition  
    GPIO.output(SDA,False)  
    GPIO.output(SCL,True)   
    GPIO.output(SDA,True)  
   

ADDRESS = 0100111 comes partly from the datasheet of PCF8574 which converts the serial data sent through I2C to parallel pin states and vice versa. The beginning 0100 is fixed but the last three bits are set through A2...0 pins of the chip. Luckily the manufacturer was kind enough to provide the circuit diagram of the display and the expander chip. Here's its essential part: 

We see all address pins are connected to VCC, so the ending of the address is 111. Now we have everything to test the code out.

Before getting into the display itself, I first wanted to try something simple. From the datasheets and the diagram above, we can see that the pin P3 of the I2C chip controls the backlight LED of the display through a transistor. So writing something like 00000000 to the chip would turn it off and 00001000 back on. Using my I2C code it looks like this:

 write_bytes([[0,0,0,0,0,0,0,0]])  #turn backlight OFF
 time.sleep(1)                     #1 sec delay, remember to import time
 write_bytes([[0,0,0,0,1,0,0,0]])  #turn backlight ON
   
 GPIO.cleanup() #Clears the pin setup on exit 

And the light turns off and on again! I guess I would qualify as an IT support person. I didn't took video this time but I'm sure you can imagine how it looks like.

So the I2C communication is up and running. In the next post I'll take this step further and actually write something on the screen.

---

Ps. Now, when I'm writing these things down, they seem and feel straightforward but it's very misleading. There are some tedious stuff that are necessary to go through, like gathering information from datasheets and internet, but typically most of the time goes to hunting down errors and mistakes. Usually stupid ones.

Like here I first thought that I don't have to read the ACK bit, because I'll notice it anyway whether the screen works or not. I'll save a bit of effort just neglecting it. As a result I spend all night (until 4 am) trying to fix my code in vain. The next morning I started to think that maybe giving the control to the slave is actually important and I got it working in less than 10 minutes.

Lesson learned: Respect the protocol.

References:

LCD 1602 datasheet

PCF8574 datasheet

I2C on Wikipedia: https://en.wikipedia.org/wiki/I²C

Python logo https://commons.wikimedia.org/wiki/File:Python_logo_and_wordmark.svg