ESP8266 as an HTTP client

I recently blogged about how to setup an ESP8266 as a file server, in particular using SPIFFs to store the files.  But how can we retrieve HTML and general files from a web server?  In this post I'll demonstrate some code to make a request for a file from a server, then save the results on the SPIFFS on the ESP8266 client.  In terms of implementation, this works very well on a NodeMCU ESP-12E board.  I've also tried it on a ESP-01S board - there is a technical issue writing to SPIFFs within a program on some of the latest versions of this board.  There is a fix however, discussed at the end of this post.

In fact the code for an ESP8266 HTTP client is surprisingly straight-forward, and modified from here and here.

#include <Arduino.h>

#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>

#include <ESP8266HTTPClient.h>

#include <WiFiClient.h>
#include <FS.h>   // Include the SPIFFS library

ESP8266WiFiMulti WiFiMulti;

void setup() {

  Serial.begin(115200);

  Serial.println();
  Serial.println();
  Serial.println();

  WiFiMulti.addAP(NETWORK_NAME, NETWORK_PASSWORD); // ADD YOUR NETWORK CREDENTIALS HERE
  
  while (WiFiMulti.run() != WL_CONNECTED) { // Wait for the Wi-Fi to connect
    delay(250);
    Serial.print('.');
  }

  Serial.println('\n');
  Serial.print("Connected to ");
  Serial.println(WiFi.SSID());              // Tell us what network we're connected to
  Serial.print("IP address:\t");
  Serial.println(WiFi.localIP());           // Send the IP address of the ESP8266 to the computer

  if (SPIFFS.begin()) {
      Serial.println("SPIFFS Active");
      Serial.println();
  } else {
      Serial.println("Unable to activate SPIFFS");
  }

}

void UpdateSPIFFSFileFromServer(String Serverpath, String filename)
{
    WiFiClient client;
    HTTPClient http;

    Serial.print("[HTTP] begin...\n");

    Serial.println("Trying to get:" + Serverpath);
    if (http.begin(client, Serverpath)) { // HTTP

      Serial.print("[HTTP] GET...\n");
      // start connection and send HTTP header
      int httpCode = http.GET();

      // httpCode will be negative on error
      if (httpCode > 0) {
        // HTTP header has been send and Server response header has been handled
        Serial.printf("[HTTP] GET... code: %d\n", httpCode);

        // file found at server
        if (httpCode == HTTP_CODE_OK || httpCode == HTTP_CODE_MOVED_PERMANENTLY) {
          String payload = http.getString();
          //Serial.println(payload);

          File f = SPIFFS.open(filename,"w");
          if (!f)
          {
            Serial.println("file open failed");
          }
          else
          {
          Serial.println("File: "+ filename +" open!");
          f.print(payload);
          f.close();  
          }
          
        } // end httpCode OK
      } //end httpCode>0
      else {
        Serial.printf("[HTTP] GET... failed, error: %s\n", http.errorToString(httpCode).c_str());
      }

      http.end();
    } // end http.begin
    else {
      Serial.printf("[HTTP} Unable to connect\n");
    }
}

    
void loop() {
  // wait for WiFi connection
  if ((WiFiMulti.run() == WL_CONNECTED)) {

    UpdateSPIFFSFileFromServer("http://192.168.0.45/tweets.txt", "/tweets.txt");
    UpdateSPIFFSFileFromServer("http://192.168.0.45/hashtags.txt", "/hashtags.txt");

  } // WL_CONNECTED
  delay(10000);
}

The setup function connects to the network and starts the SPIFFs file system.  It is crucial that SPIFFs is started properly, otherwise the ESP8266 won't be able to write to the file system.  Once connected the loop() function is called and two example function calls to UpdateSPIFFSFileFromServer are made.  This function takes two parameters; the first is the URL of the requested file, the second is the path in the local SPIFFs file system to save the retrieved file from.  In this way the files "tweets.txt" and "hashtags.txt" are requested from my local server @ 192.168.0.45, and written to the respective files locally.  This is a neat way of updating my BeemoPoops (my litter tray twitter bot!) remotely, without having to take the ESP-01S board out of it's circuit and reprogramming.  As the device is in a deep sleep mode most of the time, I can't talk to it over the WiFi - instead I have it request the files from 192.168.0.45 when it wakes up and update them.  If it can't connect, it simply uses the local file in SPIFFs.

Problems with writing to SPIFFS on the ESP-01S and how to solve

Whilst the above code works nicely on an ESP-12E board, it was failing on an ESP-01S board.  In fact, it was possibly to write to SPIFFs using the Arduino IDE (see here for details) and I could read from SPIFFs within the sketch on the ESP-01S board, but if I tried to write to a file within a sketch, it would fail, and make the file disappear.  In fact this is a problem that has been identified relatively recently for boards with PUYA flash (P25Q80) with a flash ID = 00146085.  I don't think the patch necessary has been rolled into a stable release yet, however it is straightforward to apply.

Navigate to your Arduino folder that contains the hardware descriptions and cores.  I had some difficultly finding this at first, until I realised it was located under:

C:\Users\Duncan\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0\cores\esp8266

Note, you may need to unhide hidden folders as AppData is hidden otherwise.  In this folder you'll find "Esp.cpp".  Locate the function "EspClass::flashWrite and replace it with:

 bool EspClass::flashWrite(uint32_t offset, uint32_t *data, size_t size) {
    static uint32_t flash_chip_id = 0;
        uint32_t *read_buf=NULL;
    if (flash_chip_id == 0)
        flash_chip_id = getFlashChipId();
    ets_isr_mask(FLASH_INT_MASK);
    int rc;
    uint32_t* ptr = data;
    if ((flash_chip_id & 0x000000ff) == 0x85) { // 0x146085 PUYA
        read_buf=new uint32_t [SPI_FLASH_SEC_SIZE / 4];
        if(!read_buf) return false;
        rc = spi_flash_read(offset, read_buf, size);
        if (rc != 0) {
            delete read_buf;
            ets_isr_unmask(FLASH_INT_MASK);
            return false;
        }
        for (size_t i = 0; i < size / 4; ++i) {
            read_buf[i] &= data[i];
        }
        ptr = read_buf;
    }
    rc = spi_flash_write(offset, ptr, size);
        if(read_buf!=NULL) delete read_buf;
    ets_isr_unmask(FLASH_INT_MASK);
    return rc == 0;
}




Of course you may want to make a backup of this file if you are faint of heart.  Now close the Arduino IDE and reload everything.  You'll need to trigger the compiled to build all - for me it helped to change some settings (i.e. the SPIFFs allocation) and then compile.  It should compile fine and then allows the ESP-01S to write to SPIFFs in the correct manner.

References:

https://arduino-esp8266.readthedocs.io/en/latest/esp8266wifi/client-examples.html

https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266HTTPClient/examples/BasicHttpClient/BasicHttpClient.ino