I started working on being able to access files on a MicroSD card right after my work on preemptive multitasking. I had already found a tutorial for how to get that to work and purchased the recommended card reader. However, I really needed to solder in the pin connectors to my board so that I could properly use my breadboard to connect to the reader. My soldering pencil was still on its way, so I started working on the software part of this.
The tutorial I was working from used the Arduino’s built-in SD library. OK. Step 1: Include the requisite SD.h header and see what happens. Ho… ly… cow… That one include alone increased the flash consumed by 5,930 bytes and the RAM consumed by 683 bytes. It consumed so much RAM that there wasn’t enough dynamic memory left for login authentication anymore. The system was completely unusable! Aaaaahhhhhh!!!!!!!!
OK… Breathe. I quickly revised the login algorithm (again) to split memory between stack and dynamic memory instead of trying to be all one way or all the other way. That allowed me to login again. Still, though, the sytem was now running with only 636 bytes of total dynamic memory. Of that, only 396 was actually usable because each shell uses some in order to login and parse a command line. And, if all that wasn’t bad enough, because of the way the dynamic memory is allocated and freed, I could only successfully login once. The second time around, there wasn’t enough memory available. So, I made a second change to how memory was managed during login to allow for more than one login.
RAM usage, however, was only part of the problem and, arguably, perhaps the smaller one. The increase in flash usage meant that my sketch was now consuming 49,141 of 49,152 maximum bytes. So, there was no way to add ANYTHING with things the way they were. I did have plans to remove some of the unnecessary example commands as part of my next round of work, but with that much storage taken up, I wasn’t sure that would make any difference. Something had to give here.
So, time for a web search. Now, I have had extremely bad luck with most AI chatbots for the kind of work I do. They seem to be pretty good at writing English papers, but pretty poor at embedded programming. A friend of mine, however, suggested that I try Claude. I had given it a few practice problems and found that it performed better than the others, so I decided to see if it could help me with this issue. I started by asking it why including SD.h had had such a dramatic impact on memory usage. Its answer was that using the library linked in a global object that had, among other things, a 512-byte buffer to manage the contents of blocks. No way around it and not left up to the user to decide how to manage it. Fantastic…
Claude went on, however, to suggest that I could look at the SdFat library as a possible replacement for the regular SD library. It said it could be installed from the IDE’s library manager. Well, what did I have to lose? I installed the library and included the SdFat.h header to see how things looked. The answer? Relative to baseline, it caused an increased usage of 14 bytes of RAM and 2,250 bytes of flash. Still more than I would like, but definitely MUCH better and more workable.
The downside was that this was not the end of the answer, though. Unlike the SD library which declared an instance of its state on behalf of the programmer, the SdFat library required an object to be explicitly declared. And, the size of the object? 636 bytes. Some - but not much - improvement over the 683 bytes of state in the SD library.
There was, however, one saving grace about this, though. The fact that it was a variable that I had to declare myself meant that I could declare it as part of a process instead of the global address space. I had already intended to sacrifice one process slot to construct a filesystem process. 636 bytes is too large for a single process’s stack, but I could use my back-to-back process trick again and get the filesystem process two slots. So, I would still lose some memory, but a lot less than if the whole state object had to be declared outside of a running process.
Plan in hand, I set about carving out a new filesystem process. The first step was reordering the processes to make room for a new kenel process. Change a few defines and we’re done. I wasn’t happy about the loss of a general-purpose process slot, though. I decided to make a design change. Rather than only shell commands replacing the shell process, ALL processes not explicitly specified to run in the background would replace the shell process. This guaranteed that each console would always be able to run any foreground process. The only contention for process slots would be for background processes.
Testing out the changes, though, I noticed something when I logged in and got a list of running processes: Only the USB shell was running. When fiddling with the MicroSD card reader, I had disconnected the GPIO shell, so I connected it up to see what was on the console. Nothing. And, to make matters worse, trying to provide input resulted in error message coming from the console process.
I then realized the mistake I had made. Since my recent changes reducing the stack sizes, the stacks of the shell processes were too small to accommodate ANY character arrays declared locally. Using dynamic memory was a must. So, it was a darn good thing that I found an alternative to the SD library becauase I wouldn’t be able to even login if I had to use it!
OK, logins fixed, it was time to arrange a process. I started a new library for the filesystem functionality to live in, created a main loop for it, and made the scheduler start it. So far so good. Then, I declared an instance of the SdFat SdFat variable from within the process function. To my horror, this made the OS take up even more flash. Finally, I made the process function call the begin function. This pushed the code usage over the limit. sigh…
So, I removed my largest example command from the commands library. That brought the flash usage down to about 92%. Bad, but usable at least. And I could login and confirm that everything remained functional and that the filesystem process was running. So, at least that much works.
The lack of available flash storage is a huge problem, though. My goal here is to be able to support commands compiled to Web Assembly. That means I need to develop a mechanism to be able to interpret Web Assembly binaries. At the moment, I only have 3,563 bytes to do that in. It’s not likely I’ll be successful with that little space available. Long term, my goal is to be able to remove the commands library entirely. Until then, I at least have to have the shell itself and, the ps command, and the kill command available.
I’m probably looking at having to write my own library to interface with the file system on the SD card. From the perspective of writing an OS, that’s a good thing because it will force me to develop a proper driver layer. From the standpoint of being able to make actual progress on being able to run arbitrary commands, it’s a pain in the rear.
So, I press on. I’ll have to see how far I can get with this infrastructure before having to stop and develop my own drivers and hardware abstraction layer. To be continued…