Wednesday, November 23, 2011

Ready for Christmas

One of the jobs I want to hand over to the WebCell is turning the Christmas lights & decorations on and off. I’ve been using mechanical timers, which is fine until there’s a power outage and they fall behind.

I spent three days last week creating the web pages needed to enter the Insteon control devices and the times to turn them on and off. It was very similar to the thermostat pages and I used them as a starting point. All the underlying scheduling features of the WebCell were supposed to ready, but one of the days was spent debugging new bugs the pages exposed.

I wasn’t satisfied when I was done, because I was still skipping over one of the last unimplemented features in the CellScript specification: the ability to schedule relative to sunrise and sunset. It made a lot of sense to do this now, since I would want the lights to go on about sunset. So I attacked that problem this week.

I did a web search to find the equations, and used a spreadsheet to enter test values and see all the intermediate values. The equations use a lot of trig functions, but they’re all in the C runtime library for the AVR. It took two days to get “sun-based” scheduling fully implemented, and one more day to rework the scheduling web page to accept entries. I put one set of decorations on a sunset-to-midnight schedule so I can watch and make sure it seems to be working right.

Friday, November 4, 2011

Time to Fix the DST Bug

I haven’t done any work on the WebCell since August when it took over my thermostat schedule. But Daylight Savings Time ends in a few days, so I thought I should fix a bug in the code that handles it.

I didn’t test it, but I’m pretty sure the simple way the WebCell was figuring the end of Daylight Time would behave badly. I suspect it would have spent an hour ping-ponging back and forth between Daylight and Standard Time. You can see how it would happen: at 2:00 AM it sets the clock back to 1:00 AM; then it decides Daylight Time isn’t over for another hour and sets it back to 2:00 AM, etc.

The fix wasn’t hard – just do everything in Standard Time. Instead of Daylight Time ending at 2:00 AM Daylight Time, it ends at 1:00 AM Standard Time.  All fixed. (Internally, time is kept in UTC [GMT] with time zone adjustment added to get Standard Time.)

All the parameters of Daylight Time can be changed in CellScript code, but up till now I hadn’t created a user interface to do it. I spent more time on the web page to view and edit Daylight Time settings than I did fixing the bug.

Thursday, September 22, 2011

The Scheduler

When I started this project, I envisioned the WebCell as the user interface, via web pages, for otherwise self-contained home automation devices. For example, the WebCell might pass on to a thermostat a list of temperatures and times that the user had entered, and the thermostat would handle it, even if the WebCell was subsequently disconnected. In practice, however, the devices I’ve encountered need more supervision – they need a controller to maintain the schedule.

In January 2011, I started the design of the CellScript scheduler. I wanted total flexibility, and used the calendar feature of Microsoft Outlook as a guide for creating recurring schedules. The plan made its way into the CellScript Language Reference, and from there the coding began. By the end of February, the scheduler successfully “fired an event” on time – simulated, of course.

In March I felt the scheduler was in pretty good shape. It was time to write the web pages that would collect the settings and feed them into the scheduler. To help with this I took a class at the nearby community college in cascading style sheets (CSS), which is all about making web pages look the way you want them to.

Still I found writing the web pages to be a major challenge. The pages all use AJAX techniques and YUI 3 to make them as easy to use as possible. I was creating these pages to work with an Insteon irrigation controller and a pair of Insteon thermostats (for two separate furnaces in my house). For each system I had a setup page (irrigation zones, for example), and then a settings page (such as watering time for each zone and watering schedule).

My priority was the irrigation system, because I’d already wired in the Insteon controller. Fortunately we had a very cool and wet spring in Western Washington, giving me more time to get the irrigation pages ready. In late May the WebCell succeeded in starting an irrigation cycle on schedule, which turned out to be about two weeks to spare.

The push to get irrigation running on the WebCell gave me my first real experience with CellScript (as opposed to writing test cases). I did some tweaking and bug fixing before returning to the next set of web pages, for the thermostats. On August 13, 2011, the WebCell took control of the thermostat schedule. Program size is now at 83,464 bytes. And I will be taking a break from WebCell development for a while!

Slow Progress on the List

With the production prototype in hand and working, and plenty of program space available in the XMEGA, 2010 was spent methodically working through “the list” – the CellScript Language Reference. Hardware features came first, such as ADC, DAC, PWM, and input capture (measuring the pulse width or frequency of an incoming signal). This would allow testing circuitry on the DevKit whose job was to demonstrate these features.

But the list was very long, and sometimes got longer instead of shorter. Work items included getting multiple scripts to run at once, implement the DHCP network protocol, storing important data in the internal non-volatile memory (EEPROM), adding “MIME types” to web pages – and fixing lots and lots of bugs.

The WebCell has several ways to communicate with the outside world besides Ethernet, including three serial ports, an SPI (Serial Peripheral Interface) port, and an I2C (Inter-Integrated Circuit) port. “The list” has just a few functions and properties that CellScript needs to utilize these ports – things like read(), write(), and bitRate. Implementing these for the WebCell was a clearly understood task. However, what would these do on the simulator? This question applied to all the hardware features. I spent much of August creating ways for the user to interact with simulated hardware features, allowing entry of the stuff going in, and display of the stuff coming out.

In September I started creating web pages designed to demonstrate and test hardware features. In the process I looked into YUI, the Yahoo! User Interface. The first page I made was for testing PWM (pulse-width modulation), and it used a YUI “slider” to adjust the PWM width. YUI also made it easy to have the web page contact the server in the background (in this case, the WebCell or simulator) for instant updates – a common way to make web pages more interactive called AJAX (asynchronous JavaScript and XML).

The demo web pages were helpful in completing testing of the production prototype. Satisfied that I’d found all the wiring errors, I began revision of the PCB. On September 16, 2010 I sent in the order for new PCBs, and received them October 4. When assembled and put to the test, no problems were found in the hardware design. I now had two complete working units: the production prototype, with its cuts and jumpers; and the final production model.

Using the new hardware, I set out to give the WebCell the ability to have its firmware updated over the network. Part of that process included transmitting a somewhat large firmware file from the PC to the WebCell via FTP, which turned out to expose numerous problems in previously untested FAT file code. I had had a similar experience back in July, which had got me thinking about how to test the FAT file code better.

The CellScript simulator was a great way to test and debug CellScript itself, because the same source code was used for both the simulator and the real WebCell. However, the simulator just used Windows for file operations, so it didn’t help test the FAT file system. But I realized I could change that, using raw sector read/write calls through Windows. In just a few days I was able to create a special version of the simulator, just for testing, that used my FAT file code to read & write files on an SD card on the PC.

I did a lot of testing and fixed a lot of bugs in the FAT file system, now using nothing but the simulator. In December I ended up doing a major overhaul of the whole system. And now I had a large set of tests I could run anytime to make sure the FAT file system was working right. Program size when I was finished was 70K, out of the 128K available.

Thursday, September 1, 2011

Final Hardware Design

In early 2008 Atmel announced a whole new line of microcontrollers called XMEGA that sounded pretty interesting. Compared to the existing ATmega line, they offer much higher performance and much more capability, all at around the same price or less.

As a new product, availability is an issue. I originally placed an order when they first showed up in my distributor’s catalog in September 2008, and just let it ride on backorder. The order was eventually cancelled because that specific part (ATxmega64A4) was never produced. A year later I did get my hands on one of the top-of-the-line XMEGA chips, the ATxmega256A3. This was not the chip I would use for my final hardware design, but it did give me a chance to test it out.

The XMEGA family is only available in surface mount (SMD) packages, so I mounted mine onto a DIP adapter. Then I put it on my solderless breadboard in place of the ATmega324P, wired up to an SD memory card socket and serial port. By the end of September 2009 I had my “mini-DOS” FAT/SD test program working on it.

SD Card Breadboard with XMEGA. Parts include a small 3.3V regulator, reset button, and simple RS-232 serial interface just using a 74AC04 inverter.

Also in September I started work on the hardware design for the “production” WebCell, using the ATxmega128A3. I had never used either of the radio circuits on the engineering prototype, so I decided to leave them out. In the home automation application I envisioned, the WebCell would have to connect to some other device. I had experimented with Insteon products so I knew the WebCell could be hooked up to control their devices, for example.

In November I finished the layout of the WebCell PCB. Next I started design of the “Development Kit”. This is another PCB that the WebCell can plug into that makes it easy to try out. For example, it powers the WebCell from USB, provides a serial connector, and includes a reset button.

I also designed a variation of the WebCell that I call the FileCell. This variation removes the network interface and replaces it with a USB port and battery backup. So instead of being a web server, it is more of a general-purpose controller module. It could be particularly useful for data logging because of the high storage capacity available on the microSD card. I wasn’t sure if I would ever build this variation, but I wanted to explore how it would be different from the WebCell so I could make sure the Development Kit would work with both. Because of the battery backup, I did end up changing some of the interface connector pins on the WebCell.

I had to order samples of various parts to decide which ones I wanted to use and how they would fit. I finally placed orders for production prototype PCBs on January 21, 2010. I had been planning to get PCBs for all three designs, but in the end I saw no reason to build the FileCell variation yet.

The PCBs arrived on February 8. I took them and a kit of parts to my local electronics assembly company, Printed Circuits Assembly. Both boards use chips in leadless (QFN) packages, which are really hard to solder by hand. I got the assembled units back about a week later.

WebCell on DevKit. Blue jumper wires correct some PCB layout errors, and provide test points to internal circuitry. Long tails on board-to-board connector were useful as additional test points. Micro USB connector for power is in lower right corner.

It took a few weeks to get the software ready for the production prototype and to get the prototype working. The final hitch was an error in the wiring of the Ethernet jack, requiring cuts and jumpers of the PCB traces. Once that was fixed, on February 28, 2010, the prototype began serving up web pages.

During the wait for assembly, my work on the WebCell software started bumping up against the 64K program limit of the engineering prototype. The production prototype with its 128K program size came just in time to keep me working.

Wednesday, August 31, 2011

WebCell Script File Updates

I spent the spring of 2009 working on my implementation of the FAT file system. A year earlier I had it reading files from an SD card, but now I wanted it to write to the SD card. Before the start of summer I had the essentials working – creating folders, creating files, and writing data to files.

This gave me the opportunity to relieve a long-standing annoyance. Up to now, when I wanted to test a new or updated script on the WebCell, it took a few steps:

  • Remove the microSD card from the WebCell.
  • Put the microSD card in an SD adapter.
  • Plug the SD adapter into an SD slot on my monitor.
  • Copy the test script from the PC to the microSD card.
  • “Eject” the microSD card.
  • Remove the SD adapter from the monitor.
  • Move the microSD card from the adapter back to the WebCell.

Now that the WebCell could write files itself, this process could be eliminated by sending the file to the WebCell over the network connection. And the way to do that was with FTP – File Transfer Protocol. I set to work on the implementation.

On July 9, 2009, I used FTP on Windows Explorer to copy a test script from the PC to the WebCell. This was a satisfying if not momentous milestone. Program size on the WebCell with FTP completed was at 55,510 bytes out of the 65,536 bytes available.

With the WebCell receiving files by FTP, I felt obligated to add an FTP client to the CellScript simulator. The simulator allowed me to edit scripts, so it just seemed wrong if it couldn’t send those updated scripts on to the WebCell.

Unfortunately, this would mean a lot work to do on the simulator. The simulator was written in C++ so it could share the code for CellScript with the WebCell. But I really missed the very elegant and refined environment that came with programming in Microsoft’s C# programming language. I decided I wasn’t going to take it anymore.

I split the simulator in two: a module in C++ that carried the shared CellScript code, and a module in C# that handled the user interface through Windows. Once I found out how easy it was to get the two modules to talk to each other, I was kicking myself for not doing it this way in the first place. The split and translation of the UI from C++ to C# took about a week.

Once the new simulator was working the same as before the split, I started the job of adding FTP. I created a “File Manager” similar to a stripped-down Windows Explorer. But it tracked both script files on the PC used for the simulator, and files on the WebCell using FTP. It then presented them in an integrated list showing what files were different between the two. The finishing touch was that it would monitor when changes were made to simulator script files, and could automatically copy them to the WebCell immediately.

Friday, August 26, 2011

Just What is CellScript?

I had jumped in and started implementing CellScript using a book on JavaScript I’d bought and the ECMAScript Language Specification Edition 3 as my guides. But after I had some of the basic language features working, I needed to give myself more concrete direction – it was time to define just what would be in CellScript.

First of all, CellScript is only a small subset of JavaScript. All I wanted were the basic features that any programming language has: evaluating expressions, conditional (“if”) statements, looping constructs, etc. I didn’t want to try to implement any of the special features of JavaScript. I needed to draw the line at what was in and what was out.

Second, CellScript needs to extend JavaScript so it can access and control features of the WebCell. This is done by adding “host objects”, such as the Net object for dealing with the network interface or the File object for managing files on the SD card.

In late February 2009 I created the CellScript Language Reference. My productivity had started to lag as I paused each time I finished a feature to figure out what was next – or worse, spun my wheels. Once the spec was written, I was able to charge ahead.

Thursday, August 18, 2011

CellScript and the Simulator

While serving up a static web page is an important milestone, it isn’t useful. The goal of my project is to have it present web pages as the user interface for home automation functions like thermostat settings, irrigation times, and lighting schedules. That means it needs to collect information from the user, and then do something with it. The server must be programmable.

I settled on an approach very similar to Microsoft Active Server Pages (ASP). A programming script would be embedded in each web page that would tell the server what to do; this script would be stripped out before the web page was sent on to the user’s browser. Since web pages are themselves often programmed using JavaScript, I decided to also use JavaScript to program the server. Or rather a small subset of JavaScript, which I call CellScript.

But then came the question: If the AVR is running a script, how do you debug it when it isn’t working? The answer was to create a simulator on the PC. The simulator would serve up web pages and behave just like the WebCell, but allow full debugging of the embedded CellScript with single stepping, breakpoints, and viewing variable values. On September 24, 2008, I started work on the simulator, before ever starting to implement CellScript itself.

A fundamental tenet of the simulator was that it would be built from the same C++ source code files used to implement CellScript on the WebCell. That was the only way to ensure they would work the same, and of course I didn’t want to write the code twice. For the past few years I’d used the programming language C# for all my programming on a PC. But since CellScript would be written in C++, I figured the simulator would need to be in C++ as well.

I started programming the simulator using Microsoft Foundation Classes (MFC) – a standard library for creating a Windows program (nearly obsolete now) that I’d used before. But I missed features that I had gotten used to with C#, some of which were actually part of Microsoft .NET, which is really the ultimate environment for Windows programming. Before long I switched to “managed C++”, a variation of C++ for the .NET environment.

So as I implemented CellScript, I found I did all the testing and debugging in the simulator. In late November I got a script to run for the first time on the prototype hardware. This sequence became the norm: implement new features, test and debug on the simulator, then check it out on the prototype hardware. Quite often things simply worked when transferred to the prototype, a satisfying testament to the effectiveness of the simulator. The reason for creating the simulator had not been to make development easier, but that turned out to be a huge benefit.

The prototype hardware used the ATmega324P MCU, which has 32K of program space. Of that, 18K of program space was being used when the basic web server functionality was first working (before starting on CellScript). At the start of 2009, my work on CellScript had filled the entire 32K, and I still had a long way to go. I switched over to a newly built-up prototype with the ATmega644P, giving me 64K of program space. This also increased RAM size to a whopping 4K (from 2K).

Sunday, August 14, 2011

Interface to an SD Card

In February 2008 I started a related project: interfacing an AVR to an SD memory card. I was using an ATmega324P MCU on a solderless breadboard. It was wired to an SD card “breakout board” from SparkFun Electronics, and a serial port for connecting to a PC running HyperTerminal.

It didn’t take long to get basic communication with the SD card working, but to make sense of the card contents I had to implement the FAT file system. The idea of doing this was an interesting flashback, since I wrote the original FAT12 file system in 8086 assembly language almost 30 years ago. Now I was using the published Microsoft specification to implement FAT12, FAT16, FAT32, and long file names, thankfully in C++ rather than assembly language.

In March I was working (at least some) on the FAT file system almost every day. By the end of the month I had sort of a mini DOS command prompt working, with commands like DIR, CD, and TYPE. I hadn’t attempted to write to the SD card yet.

I didn’t get much more done until June, when I started pushing harder on the AVR network software. I hadn’t worked on this since last August, so some reviewing was necessary to get back into the swing of things. But by the end of the month I had the “Easy Ethernet AVR” module successfully serving up a minimal web page to my PC. The module does not have an SD card (or any other storage), so there was no real content, but it demonstrated success with IP, TCP, and HTTP network protocols.

I needed hardware that combined Ethernet and SD Card, and I set about designing it in July. This design included sockets for both a standard SD Card and a microSD card. My vision for this project was that it would be a home automation controller, communicating by ZigBee radio to other devices around the house. To that end, my engineering prototype included two radio systems: a socket for a MaxStream (now Digi) XBee radio module, and a radio circuit built around the Atmel AT86RF231 ZigBee-compatible transceiver chip. I also switched to a smaller and simpler Ethernet controller chip, the Microchip ENC28J60.

With the design complete and printed circuit board (PCB) laid out, I ordered boards from PCB Fab Express on August 1st. While they were in process I improved my SD card & FAT file system software so it could write to the card as well as read it, which didn’t take long. I also revised the software so it will work with new hardware, including drivers for the alternate Ethernet chip I was using. And of course I ordered the parts I’d need, from my favorite distributor, Digi-Key.

Engineering Prototype. At upper left is the discrete radio circuit, without the antenna jack installed. Below that is the socket for the XBee radio module. At right middle is an ATmega644P MCU. The first unit was built with the pin-compatible ATmega324P, but the program grew beyond its 32K program size limit.

The prototype boards arrived in about three weeks, and I assembled one right away. Within a few days I had it serving up a fake web page (not yet reading it from the SD card). The network protocol analyzer WireShark proved invaluable in getting this working. A few weeks later, on September 11, 2008, the prototype was serving up web pages from the SD card on request – a major milestone.

Saturday, August 13, 2011

Development of the WebCell Begins

On April 20, 2007 I started my project to make a web server based on an Atmel AVR microcontroller. Let me start catching up on this story.

The idea that it was possible to build a web server around a microcontroller unit (MCU) came from a series of articles written by Peter Best that appeared in Nuts and Volts magazine in 2006. These articles discussed network protocols, and also led me to EDTP Electronics and their “Easy Ethernet” modules for experimenters.

The “Easy Ethernet” line was available with either a Microchip PIC or Atmel AVR MCU. I had used the PIC in the distant past, but switched to the AVR once I heard about it – I consider it vastly superior. So my module has the Atmel ATmega16 MCU, which has 16K of program space and 1K of RAM. The Ethernet chip on the module is the Realtek RTL8019AS.

Easy Ethernet AVR module from EDTP Electronics. The shiny block at the upper left is the Ethernet jack. Strangely, the module did not come with a connector for Atmel’s standard debugging interface (JTAG), so I had to patch into the correct pins using the “squid”.

I spent about seven days working on the project over the next six weeks. I was writing basic drivers for the Ethernet chip and implementing the Address Resolution Protocol (ARP) as my first test. On June 3rd I hit my first milestone, verifying for the first time the module could send and receive Ethernet frames: it successfully made an ARP request and read the response.

I continued to work on the project a little on and off until later that summer, when I dropped it to do other things.