WebCell Script File Updates

I spent the spring of 2009 working on my implementation of the FAT file system. A year earlier I had it reading files from an SD card, but now I wanted it to write to the SD card. Before the start of summer I had the essentials working – creating folders, creating files, and writing data to files.

This gave me the opportunity to relieve a long-standing annoyance. Up to now, when I wanted to test a new or updated script on the WebCell, it took a few steps:

• Remove the microSD card from the WebCell.
• Put the microSD card in an SD adapter.
• Plug the SD adapter into an SD slot on my monitor.
• Copy the test script from the PC to the microSD card.
• “Eject” the microSD card.
• Remove the SD adapter from the monitor.
• Move the microSD card from the adapter back to the WebCell.

Now that the WebCell could write files itself, this process could be eliminated by sending the file to the WebCell over the network connection. And the way to do that was with FTP – File Transfer Protocol. I set to work on the implementation.

On July 9, 2009, I used FTP on Windows Explorer to copy a test script from the PC to the WebCell. This was a satisfying if not momentous milestone. Program size on the WebCell with FTP completed was at 55,510 bytes out of the 65,536 bytes available.

With the WebCell receiving files by FTP, I felt obligated to add an FTP client to the CellScript simulator. The simulator allowed me to edit scripts, so it just seemed wrong if it couldn’t send those updated scripts on to the WebCell.

Unfortunately, this would mean a lot work to do on the simulator. The simulator was written in C++ so it could share the code for CellScript with the WebCell. But I really missed the very elegant and refined environment that came with programming in Microsoft’s C# programming language. I decided I wasn’t going to take it anymore.

I split the simulator in two: a module in C++ that carried the shared CellScript code, and a module in C# that handled the user interface through Windows. Once I found out how easy it was to get the two modules to talk to each other, I was kicking myself for not doing it this way in the first place. The split and translation of the UI from C++ to C# took about a week.

Once the new simulator was working the same as before the split, I started the job of adding FTP. I created a “File Manager” similar to a stripped-down Windows Explorer. But it tracked both script files on the PC used for the simulator, and files on the WebCell using FTP. It then presented them in an integrated list showing what files were different between the two. The finishing touch was that it would monitor when changes were made to simulator script files, and could automatically copy them to the WebCell immediately.

Just What is CellScript?

I had jumped in and started implementing CellScript using a book on JavaScript I’d bought and the ECMAScript Language Specification Edition 3 as my guides. But after I had some of the basic language features working, I needed to give myself more concrete direction – it was time to define just what would be in CellScript.

First of all, CellScript is only a small subset of JavaScript. All I wanted were the basic features that any programming language has: evaluating expressions, conditional (“if”) statements, looping constructs, etc. I didn’t want to try to implement any of the special features of JavaScript. I needed to draw the line at what was in and what was out.

Second, CellScript needs to extend JavaScript so it can access and control features of the WebCell. This is done by adding “host objects”, such as the Net object for dealing with the network interface or the File object for managing files on the SD card.

In late February 2009 I created the CellScript Language Reference. My productivity had started to lag as I paused each time I finished a feature to figure out what was next – or worse, spun my wheels. Once the spec was written, I was able to charge ahead.

CellScript and the Simulator

While serving up a static web page is an important milestone, it isn’t useful. The goal of my project is to have it present web pages as the user interface for home automation functions like thermostat settings, irrigation times, and lighting schedules. That means it needs to collect information from the user, and then do something with it. The server must be programmable.

I settled on an approach very similar to Microsoft Active Server Pages (ASP). A programming script would be embedded in each web page that would tell the server what to do; this script would be stripped out before the web page was sent on to the user’s browser. Since web pages are themselves often programmed using JavaScript, I decided to also use JavaScript to program the server. Or rather a small subset of JavaScript, which I call CellScript.

But then came the question: If the AVR is running a script, how do you debug it when it isn’t working? The answer was to create a simulator on the PC. The simulator would serve up web pages and behave just like the WebCell, but allow full debugging of the embedded CellScript with single stepping, breakpoints, and viewing variable values. On September 24, 2008, I started work on the simulator, before ever starting to implement CellScript itself.

A fundamental tenet of the simulator was that it would be built from the same C++ source code files used to implement CellScript on the WebCell. That was the only way to ensure they would work the same, and of course I didn’t want to write the code twice. For the past few years I’d used the programming language C# for all my programming on a PC. But since CellScript would be written in C++, I figured the simulator would need to be in C++ as well.

I started programming the simulator using Microsoft Foundation Classes (MFC) – a standard library for creating a Windows program (nearly obsolete now) that I’d used before. But I missed features that I had gotten used to with C#, some of which were actually part of Microsoft .NET, which is really the ultimate environment for Windows programming. Before long I switched to “managed C++”, a variation of C++ for the .NET environment.

So as I implemented CellScript, I found I did all the testing and debugging in the simulator. In late November I got a script to run for the first time on the prototype hardware. This sequence became the norm: implement new features, test and debug on the simulator, then check it out on the prototype hardware. Quite often things simply worked when transferred to the prototype, a satisfying testament to the effectiveness of the simulator. The reason for creating the simulator had not been to make development easier, but that turned out to be a huge benefit.

The prototype hardware used the ATmega324P MCU, which has 32K of program space. Of that, 18K of program space was being used when the basic web server functionality was first working (before starting on CellScript). At the start of 2009, my work on CellScript had filled the entire 32K, and I still had a long way to go. I switched over to a newly built-up prototype with the ATmega644P, giving me 64K of program space. This also increased RAM size to a whopping 4K (from 2K).

Interface to an SD Card

In February 2008 I started a related project: interfacing an AVR to an SD memory card. I was using an ATmega324P MCU on a solderless breadboard. It was wired to an SD card “breakout board” from SparkFun Electronics, and a serial port for connecting to a PC running HyperTerminal.

It didn’t take long to get basic communication with the SD card working, but to make sense of the card contents I had to implement the FAT file system. The idea of doing this was an interesting flashback, since I wrote the original FAT12 file system in 8086 assembly language almost 30 years ago. Now I was using the published Microsoft specification to implement FAT12, FAT16, FAT32, and long file names, thankfully in C++ rather than assembly language.

In March I was working (at least some) on the FAT file system almost every day. By the end of the month I had sort of a mini DOS command prompt working, with commands like DIR, CD, and TYPE. I hadn’t attempted to write to the SD card yet.

I didn’t get much more done until June, when I started pushing harder on the AVR network software. I hadn’t worked on this since last August, so some reviewing was necessary to get back into the swing of things. But by the end of the month I had the “Easy Ethernet AVR” module successfully serving up a minimal web page to my PC. The module does not have an SD card (or any other storage), so there was no real content, but it demonstrated success with IP, TCP, and HTTP network protocols.

I needed hardware that combined Ethernet and SD Card, and I set about designing it in July. This design included sockets for both a standard SD Card and a microSD card. My vision for this project was that it would be a home automation controller, communicating by ZigBee radio to other devices around the house. To that end, my engineering prototype included two radio systems: a socket for a MaxStream (now Digi) XBee radio module, and a radio circuit built around the Atmel AT86RF231 ZigBee-compatible transceiver chip. I also switched to a smaller and simpler Ethernet controller chip, the Microchip ENC28J60.

With the design complete and printed circuit board (PCB) laid out, I ordered boards from PCB Fab Express on August 1st. While they were in process I improved my SD card & FAT file system software so it could write to the card as well as read it, which didn’t take long. I also revised the software so it will work with new hardware, including drivers for the alternate Ethernet chip I was using. And of course I ordered the parts I’d need, from my favorite distributor, Digi-Key.

Engineering Prototype. At upper left is the discrete radio circuit, without the antenna jack installed. Below that is the socket for the XBee radio module. At right middle is an ATmega644P MCU. The first unit was built with the pin-compatible ATmega324P, but the program grew beyond its 32K program size limit.

The prototype boards arrived in about three weeks, and I assembled one right away. Within a few days I had it serving up a fake web page (not yet reading it from the SD card). The network protocol analyzer WireShark proved invaluable in getting this working. A few weeks later, on September 11, 2008, the prototype was serving up web pages from the SD card on request – a major milestone.

Development of the WebCell Begins

On April 20, 2007 I started my project to make a web server based on an Atmel AVR microcontroller. Let me start catching up on this story.

The idea that it was possible to build a web server around a microcontroller unit (MCU) came from a series of articles written by Peter Best that appeared in Nuts and Volts magazine in 2006. These articles discussed network protocols, and also led me to EDTP Electronics and their “Easy Ethernet” modules for experimenters.

The “Easy Ethernet” line was available with either a Microchip PIC or Atmel AVR MCU. I had used the PIC in the distant past, but switched to the AVR once I heard about it – I consider it vastly superior. So my module has the Atmel ATmega16 MCU, which has 16K of program space and 1K of RAM. The Ethernet chip on the module is the Realtek RTL8019AS.

Easy Ethernet AVR module from EDTP Electronics. The shiny block at the upper left is the Ethernet jack. Strangely, the module did not come with a connector for Atmel’s standard debugging interface (JTAG), so I had to patch into the correct pins using the “squid”.

I spent about seven days working on the project over the next six weeks. I was writing basic drivers for the Ethernet chip and implementing the Address Resolution Protocol (ARP) as my first test. On June 3rd I hit my first milestone, verifying for the first time the module could send and receive Ethernet frames: it successfully made an ARP request and read the response.

I continued to work on the project a little on and off until later that summer, when I dropped it to do other things.