Discovering Arduino and fascinating world of electronics

P4A – PHP for Arduino part 2

Posted: May 23rd, 2011 | Author: | Filed under: arduino, For beginners, howto | Tags: , , , , , , , , , , , , | 1 Comment »

In the previous post we’ve finished with reading files from SD card. Now we have to pass them through our PHP :) and send the results to the client. To make whole process more simple, we’re assuming each of the files is known to us. This means, we’re registering every file (URL) with addCommand. How it should work? The idea is to keep our functions in the sketch code, whose results should be put into selected places of HTML code. In other words, we want to have HTML file with such a code snippet:

<p>
Sensor 1 reading: MAGIC1<br/>
Sensor 2 reading: MAGIC2<br/>
</p>

Our parser’s goal is to replace MAGIC1 and MAGIC2 with values returned by the functions from the sketch.

It’s best to begin from the magic!

How to write a code in HTML so to tell our parser to substitute it with another text. For simplification we’ll assume the following pattern: #{X} will be substituted with the value returned from appropriate functioni. X is an one-character mnemonic denoting the function we want to invoke. The functions must have a specified definition and mustn’t receive any parameters. Why? Receiving arguments will make the parser unnecessarily complicated and we rather don’t need it now. We’ve already chosen the method of writing entries to HTML code. Now, a question arises, how to pass the returned value from our function? We’re assuming the example function must have the following definition:

void timeReport(char *buf) { itoa(millis()/1000, buf, 10); };

It doesn’t return any values (void), but receives a pointer to a text buffer. In that buffer it has to put its result, and after that, the result will be put in a proper place in the HTML code. The function must take care of the buffer in order not to overload it. Its length is declared in P4A_MAX_FUNCTION_RESPONSE_LENGTH. As you can see, the function above returns the number of seconds elapsed from turning on or resetting the Arduino. How to parse a file? Thanks to our “magic” marker it’s quite easy. We’re reading the file by one character. If we come across ‘#’ character, then we’re reading the next character and checking, if it’s ‘{‘. If does, we have the marker sequence. Until we come across ‘#’, all the characters are sent to the buffer, which will be eventually sent to the web browser. If the next character after ‘#’ isn’t the opening bracket (‘{‘), then both characters (‘#’ and the following) are sent to browser. In case the next character is a bracket, then we’re reading the next character – it’s our mnemonic! After that, a function is invoked, according to the mnemonic, and the result is sent to the web browser. Afterwards, we’re reading the file until the closing bracket (‘}’), then continuing scanning for the next ‘#’. Now it only remains for us to assign functions to mnemonics. For this, we’ll be using

Array of pointers to functions

Let’s talk about slightly more complex topic – pointers to functions. Basically, after compilation, a function is an address at which its code is located along with a contract describing what kind of data is passed to the function and how values are returned from it. If the contract (list of parameters and returned value) is the same for many functions, such a functions can be stored in a form of pointer and kept in an array. Then we are able to call such functions by using the pointer, without a need to know its name. This is what will be serving us as a mechanism translating mnemonics to called functions. Now we know why all of our functions must have the same interface/contract (as we already assigned, it will be void, regarding returned data type and char * as an argument) – thanks to this we are able to keep them in a single array with single-lettered index. But, starting from the beginning:

void (*_fcts['z'-'a'])(char *);

We’re defining an array of pointers to functions. The leading “void” denotes type of data returned by the function, that in brackets means what arguments is expected by the function. Inside is the declaration of an array. It’s name 'z'-'a' can appear strange to you, but in this context the characters are treated as numbers by the compiler. So, from ‘z’ code we’re subtracting ‘a’ code. The difference is the letter count. Thanks to it we have an array which is able to store one pointer for each small letter in Latin alphabet (or rather in ASCII charset). If we have any letter code, it’s enough to subtract the letter code from it and we’ll get the index from the array. Let’s look:

				if (_fcts[c[0]-'a'] == NULL) {
					bufferedSend(server,"n/a");
					continue;
				} else {
					//Call function from table
					_fcts[ c[0]-'a'](buf);
					//Write response to client
					bufferedSend( server, buf );
				}

c[0] contains a character of our mnemonic. If the array don’t have any value (i.e. NULL value) at c[0] - 'a' index, then in HTML code n/a is put – that’s our method of indicating wrong mnemonics. 'a' has a value of 97 in ASCII. If our mnemonic is also 'a' 97-97 = 0, so it’s the first element of the array. If the mnemonic is 'b', then 'b'-'a' = 98 – 97 = 1 so the second element of the array. It would be useful to know if the mnemonic is in range, otherwise we could try to call a function with a random address (when the array index is out of range, RAM memory out of the array will be read and the processor will try to interpret a value at this address as an address of a function to invoke – easy way to crash the program). In the code above you can also see, how to call a function from the array:

_fcts[ c[0]-'a'](buf);

We’re using status variable for file analysis. It indicates the status of the parser. It can be:

  • P4A_PARSE_REGULAR – searching for ‘#’
  • P4A_PARSE_HASH – waiting for the opening bracket (‘{‘)
  • P4A_PARSE_COMMAND – searching for mnemonic

Thanks to the status it’s easier to control, what our parser is doing. According to current status and next letter we can make decision, what to to afterwards. For the interested ones, here is whole parseP4A program code:


//Reads HTML file, parses looking for our macro and sends back to client
int parseP4A( char * filename, WebServer &server ) {
	//simple status
	short int STATUS = P4A_PARSE_REGULAR;
	char c[2];
	c[1] = 0;

	//buffer to hold response from functions - there is no boundary checking, so
	//function has to not overwrite data
  char buf[P4A_MAX_FUNCTION_RESPONSE_LENGTH];

  if (! file.open(&root, filename, O_READ)) {
		return -1;
  }
  while ((file.read(c,1) ) > 0) {
		if (STATUS == P4A_PARSE_REGULAR && c[0] == '#')
		{
			//hash was found we need to inspect next character
			STATUS = P4A_PARSE_HASH;
			continue;
		}

		if (STATUS == P4A_PARSE_HASH) {
			if (c[0] == '{') {
				//go to command mode
				STATUS = P4A_PARSE_COMMAND;
				continue;
			} else {
				//fallback to regular mode, but first send pending hash
				STATUS = P4A_PARSE_REGULAR;
				bufferedSend(server, "#");
			}

		}

		if (STATUS == P4A_PARSE_COMMAND) {
			if(c[0] == '}') {
				STATUS = P4A_PARSE_REGULAR;
				continue;
			};
		  if (c[0] >= 'a' && c[0] <='z') {
				//Command found
				if (_fcts[c[0]-'a'] == NULL) {
					bufferedSend(server,"n/a");
					continue;
				} else {
					//Call function from table
					_fcts[ c[0]-'a'](buf);
					//Write response to client
					bufferedSend( server, buf );
				}
			}

		}

		if (STATUS == P4A_PARSE_REGULAR)
			bufferedSend(server, c);
  }

  //force buffer flushing
  flushBuffer(server);
	file.close();
  return 0;
}

P4A – PHP for Arduino in action

Let’s assume we want to make a nice virtual barometer, but showing real air pressure. We’ll take advantage of BMP085 – it’s a breakout board for pressure and temperature sensor made by SparkFun. We’re going to have the following result:

Virtual barometer's appearance

Virtual barometer's appearance

The hand should point to value read from the sensor and the foretasted weather symbol should change according to air pressure changes. How do we get on with that, when, as it was said in the previous part, WWW server on Arduino is not a best idea (loading many files – pictures(!) simultaneously)? So, if everything is going to be available from the web, why don’t we make the static elements also available from there? These elements are: the barometer’s dial, hand image and weather symbols. On Arduino, on SD card, there’s a HTML file. In the sketch we’re writing a function calling the parser for this file:

void index(WebServer &server, WebServer::ConnectionType type, char *, bool){
  server.httpSuccess();
  if (!parseP4A("BARO.HTM", server)) {
		Serial.println("P4A: -1");
	}
};

parseP4A is a function parsing a given file and sending the result using Webdiuno object. It only remains to register our function as the default command:

webserver.setDefaultCommand(&index);

The HTML itself uses JavaScript and canvas object to draw the dial. This is done by draw function, receiving pressure in hectopascals as an argument. When the site is ready (this means, has been loaded), we’re calling draw function by onload attribute and the pressure is passed by our P4A:

<body onload="draw(#{p});"

p mnemonic has to be assigned with a proper function. In the sketch, in setup, we’re setting a function for p:

 _fcts['p'-'a'] = pressureReport;

And the pressureReport itself looks like:

void pressureReport(char *buf) {
	bmp085_read_temperature_and_pressure (&temperature, &pressure);
	itoa(pressure/100.0, buf, 10);
	Serial.print ("temp & press: ");
	Serial.print (temperature/10.0);
	Serial.print(" ");
	Serial.println (pressure/100.0);

};

Serial is used for checking, if the values are according to our expectations and there isn’t any impact on the barometer’s operation. bmp085_read_temperature_and_pressure it’s a BMP085 handling function, taken from a finnish blog. The whole code is available for download here. It’s a sketch powering our server plus HTML file and graphics. The dial, graphics and HTML/JS code was made bySprae.

Installation

Download, unpack in sketchbook. Content of html subdirectory has to be put into the main catalog on SD card, then insert the card into the shield. Correct the sketch by specifying proper MAC and IP address. After opening the main site in our browser, we should see the barometer, supposing we have BMP085 ;)

Some annotations at the end

The code is beta version :) It means it worked for me, during the tests I performed and maybe there’s some bugs (and probably that’s true) I haven’t ever imagined :) The code needs to be rearranged, e.g. functions related to buffered output should be moved to Webdiuno code. I’m planning to make it and submit all the changes to Webduino developers. Maybe some of them will be included in the library. I’m curious of your opinion. Does something like P4A have any sense and could it be useful?


P4A – PHP for Arduino part 1

Posted: May 22nd, 2011 | Author: | Filed under: arduino, For beginners, howto | Tags: , , , , , , , , , , , , | 1 Comment »

If you already know Arduino stuff a bit, maybe you want to connect it with a network somehow. It could be some network-enabled home automation solution or a set of sensors sending data to a database. So, here comes the Ethernet Shield to help you.

First, a bit of history. Initially, the Ethernet Shield was compatible with the small Arduino. Why not with Mega? It’s because for communication with W5100 IC, a heart of the shield, SPI protocol is used. In Arduino Mega SPI occupies a different set of digital outputs (not 10, 11, 12 and 13). Of course, there’s a workaround for this – wiring the shield up with cables.

That involves the older shields. The ones which are currently sold by Nettigo are compatible both with small Arduinos (UNO, Duemilanove) and Mega (with ATmega1280 and ATmega2560 microcontrollers). How to recognize such a shield? If there’s a microSD slot – it’s the newer version.

Ethernet Shield with microSD slot

Ethernet Shield with microSD slot

 

 

W5100 – the IC powering Ethernet Shield is a chip made by Korean-based WIZnet. The difference between it and other Ethernet controllers is TCP/IP stack implementation. In W5100, the stack is implemented directly in the chip. What it means for an ordinary user? The communication library uses less RAM and occupies less space on flash memory in compare to chips with no TCP/IP stack.

How to take advantage of Ethernet Shield?

There’s plenty of examples on the Internet, how to make websites displaying data from Arduino. However, if you have any bigger experience with Arduino programming, you probably know that every single text string, even the one defined in the program code, needs some amount of space in RAM memory. And, of course, there always lacks of RAM memory.

Let’s check out the official example from the Arduino Tutorial. Code:

 client.println("HTTP/1.1 200 OK");
 client.println("Content-Type: text/html");
 client.println();

It will take 40 bytes of RAM (15 chars in HTTP… + trailing zero and 23 in Conte… + trailing zero). It’s simple to imagine, how important it is when we have only 2kB to use. The HTML website shouldn’t be too complex.
Fortunately, there’s a soution – storing the strings in Flash memory. It can lower the usage of RAM memory, but sometimes by adding some code. Accessing strings defined in this way is possible only with a special macro and the compiler wouldn’t let us to use this macro in function char * as a parameter. What’s more, any change in the HTML code forces us to recompile whole sketch and upload it to Arduino.

Wait, the Ethernet Shield has a microSD slot, hasn’t it? Couldn’t we take advantage of it somehow? So, it’s possible, but requires a bit of work.

First – the Ethernet Shield has to be configured for work in a network – this means setting the MAC and IP addresses. This can be done like that:

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 192,168,1, 177 };

Ethernet.begin(mac, ip);

MAC address is given on a sticker on the back side of the shield. IP address depends on the network’s configuration. The sequence Ethernet.begin will work only in a local network, this means, when all of the IP addresses belong to the same subnet. If the shield should connect with hosts from other networks (either when operating as a client or as a server) you must specify one more parameter – an array containing 4 numbers – the IP address of default gateway (Yours router IP address). More on the Ethernet Library reference page.

Most of the blocks are ready. The Ethernet library found in Arduino IDE is not the best solution. It simplifies creating, for example, TCP server, but the HTTP server is a few steps away. Thus, the Webduino will be very useful. It’s a HTTP server based on the Ethernet library, so we have most of work done.

Normally, we would download it from there and unpack to sketchbook/libraries, but now we are going to modify Webduino a little, the final version will be available at the end of this mini tutorial, so for now, you needn’t to install anything.

Webduino – how to begin?

At first – we have to know that with this:

void addCommand(const char *verb, Command *cmd);

we are able to register any function invoked every time when a URL matches verb. Example:

webserver.addCommand("blob.htm", &blob);

If the URL begins with blob.htm, then the function blob will be invoked. The function has to receive parameters according to the definition in Command:

typedef void Command(WebServer &server, ConnectionType type,

                       char *url_tail, bool tail_complete);

server is a WebServer object, for which the method was invoked, type – type of request (INVALID, GET, HEAD, POST), url_tail the remaining part of matched URL address (after blob.htm). If the URL has to be truncated because of too small Webduino buffer, the last parameter tail_complete will have false value.

OK, but it’s not very convenient to register every single function, especially when we want to use data from SD card, which we don’t know. Now, setFailureCommand will be useful to register in our code a function invoked, when none of the functions defined with addCommand was invoked.

Now, it only remains to check if url_tail is a name of a file located on the SD card (because any matching didn’t occur, url_tail contains full URL address including first character ‘/’). If the file doesn’t exist, we’re displaying HTTP 404 error, if does – sending it to the client.

How to read a file from SD card?

Like with HTTP server, we don’t have to do everything by ourselves. A good help will be a tutorial prepared by Limor Fried – a.k.a. LadyAda. She utilizes the SdFatLib library, supporting FAT16 i FAT32 file systems (commonly found on SD and SDHC cards).

Initialization code and file handling was basically taken from the tutorial. I will describe here the fetchSD function, registered by setFailureCommand. Its objective is to find a file on the card and send it to web browser:

P(CT_PNG) = "image/png\n";
P(CT_JPG) = "image/jpeg\n";
P(CT_HTML) = "text/html\n";
P(CT_CSS) = "text/css\n";
P(CT_PLAIN) = "text/plain\n";
P(CT) = "Content-type: ";
P(OK) = "HTTP/1.0 200 OK\n";
void fetchSD(WebServer &server, WebServer::ConnectionType type, char *urltail, bool){
	char buf[32];
	int16_t  readed;

	++urltail;
	char *dot_index; //Where dot is located
	if (! file.open(&root, urltail, O_READ)) {
		//Real 404
		webserver.httpNotFound();
  } else {
	if (dot_index = strstr(urltail, ".")) {
		++dot_index;
		server.printP(OK);
		server.printP(CT);
		if (!strcmp(dot_index, "htm")) {
				server.printP(CT_HTML);

		} else if (!strcmp(dot_index, "css")) {
				server.printP(CT_CSS);

		} else if (!strcmp(dot_index, "jpg")) {
				server.printP(CT_JPG);

		} else {
				server.printP(CT_PLAIN);
		}
		server.print(CRLF);
	}
	readed = file.read(buf,30);
	while( readed > 0) {
		buf[readed] = 0;
		bufferedSend(server,buf,readed);
		readed = file.read(buf,30);
	}
	flushBuffer(server);
	file.close();
	}
}

At the very beginning we have a few constants stored in Flash memory.

P(CT_PNG) = "image/png\n";

P macro is a part of Webduino and serves for storing strings in Flash and not in RAM. And the constants are names of various content types, so called MIME types. What’s going on? The browser don’t know what data is going to be sent by the server. It finds it out from Content-Type header, which will be described after a while.

Next, we’re getting rid of the slash: ++urltail;, after that we’re trying to open a file on SD card – in case of failure – displaying HTTP 404 (Not Found) error:

	if (! file.open(&root, urltail, O_READ)) {
		//Real 404
		webserver.httpNotFound();
  } else {

and in case of success, we will try to read it in else and send it to the client.

Now some annotations. First of all – SdFatLib only supports short filenames in 8.3 format. If you use longer names (which FAT32 allows), you must remember that the name visible for SdFatLib is different than the one you see in file manager. When you create index.html file (four characters in the extension), the name for SdFatLib would be ind~1.htm. Even after renaming it to index.htm, the catalog entry will be still in extended form. You have to delete the file and create it again with a new name in 8.3 format.

The second annotation – for obvious reasons we won’t worry about the subdirectories. We’re supposing all of files are in the root directory. Maybe in further versions I will add some support for more complex directory trees.

OK, let’s go back to the code fetchSD. If we succeeded to open the file, we’ll search for a dot in its name and check if the rest (after the dot) matches any of the known extensions. This is because it’s not enough to send the data itself to the client. A header with a proper Content-Type must be sent. Otherwise, the data could be interpreted wrong by the browser.

Speaking of it – how an response from a web server should look like. It’s divided into two parts. The first are so called headers. Web browsers treats everything at the beginning as headers, until it find a blank line (the lines are separated by CR LF). The rest is the answer itself. It depends on the headers, how it will be interpreted. The server can help web browsers by setting headers describing the type of data:

Content-type: text/html

Before the colon there is a name of the header (in this case, Content-Type) and after, its value. Here are used so called MIME types. For example image/png for PNG image, image/jpg for JPG and text/html for HTML.

An obligatory header is status – tells, if a client’s request was handled or an error or redirection occurred. HTTP/1.0 200 OK means everything is OK. At the first place is the protocol and its version (HTTP, version 1.0), next the 200 code – which means everything was done properly. Other frequently used codes are 404 – content not found, 301 and 302 – redirections.

Knowing that, we’ll try to recognize the file extension and send appropriate header:

	if (dot_index = strstr(urltail, ".")) {
		++dot_index;
		server.printP(OK);
		server.printP(CT);
		if (!strcmp(dot_index, "htm")) {
				server.printP(CT_HTML);

		} else if (!strcmp(dot_index, "css")) {
				server.printP(CT_CSS);

		} else if (!strcmp(dot_index, "jpg")) {
				server.printP(CT_JPG);

		} else {
				server.printP(CT_PLAIN);
		}
		server.print(CRLF);

Functions with big P at the end don’t expect char *, but const prog_uchar *, which is data type strings stored in flash.

Now we have already the HTTP header (ended with a blank line server.print(CRLF)), so it’s time to send data:

	readed = file.read(buf,30);
	while( readed > 0) {
		buf[readed] = 0;
		bufferedSend(server,buf,readed);
		readed = file.read(buf,30);
	}
	flushBuffer(server);
	file.close();

We’re reading by 30 bytes, sending to client through a function which buffers sent data. What for? If we use the simplest method and send data by one byte, then every single character will be in a separate TCP packet . Very (really, very) ineffective solution. Simply server.write sends the data instantly. We could read file by bigger chunks and send them directly to client. Ability to read file by one char and send response in bigger chunks will be used in second part of this tutorial.

For that reason I wrote bufferedSend function, receiving WWW server object, pointer to data buffer and size of the buffer as it’s parameters. Why don’t use functions returning the size of a string, like strlen? Because it will work only with text data. When the data is binary (like images), the a character denoting end of the text string could appear in legal data stream.

In C and C++ the end of a text string is 0 (not the digit 0, but a byte with value of 0). If in our data stream zeros could appear, all the functions provided by the standard library would be useless.

For that reason we need to define the amount of the data sent to the buffer.

Basically, that’s all. We already have WWW server sending data from SD card.

Does it have any sense?

After a couple of tests with more complex websites (not only one HTML file, but some CSSes and images) you will realize that this solution has certain limits. Arduino is single-threaded, so every element from our site is downloaded separately – we need to finish sending first file before sending another. This means, from the user’s point of view, the site loads slowly if we need to fetch HTML, some CSS, images.

So, how can this be useful? Arduino can present data collected with sensors in more friendly form if the limit of RAM memory could be bypassed. Keeping HTML code on SD card we are bypassing it. But how to put data collected by Arduino sensors to HTML page on SD?

We need something that will allow us to inject data to HTML between ‘reading’ and sending’. Something like:

PHP for Arduino

OK, it’s a bit exaggerated :) We need something more like templates not full PHP, but at it’s beginning PHP also wasn’t a miracle :)

How to make such a parser (and full sketch code) – in the next part. Stay tuned.