Cloud-to-Cloud: OnlineOffline Test Fails with willReportState: false - "Device is not online" error

[Wasi78]

Hello everyone,

I'm working on a cloud-to-cloud Smart Home Action for a device with willReportState: false, but I'm consistently running into an issue with the OnlineOffline test in the Test Suite.

The device is a light that supports the OnOff and Brightness traits. I have intentionally set willReportState to false because the device state will be polled using QUERY intents.

My SYNC response is structured as follows:

{
  "requestId": "test-query-gate-001",
  "payload": {
    "agentUserId": "92",
    "devices": [
      {
        "id": "1337",
        "type": "action.devices.types.LIGHT",
        "traits": [
          "action.devices.traits.OnOff",
          "action.devices.traits.Brightness"
        ],
        "name": {
          "defaultNames": [
            "Cupboard Light"
          ],
          "name": "Cupboard Light",
          "nicknames": [
            "Cupboard Light"
          ]
        },
        "willReportState": false,
        "roomHint": "Meeting Room",
        "deviceInfo": {
          "manufacturer": "Kluger Inc",
          "model": "Model-2",
          "hwVersion": "1.0",
          "swVersion": "1.0"
        },
        "customData": {
          "internalDeviceType": "2"
        }
      }
    ]
  }
}

When the Test Suite sends a QUERY request, my fulfillment responds correctly with the online state. For example, here is a valid QUERY response from my service:

{
  "requestId": "XXXXXXXXXXXXX",
  "payload": {
    "devices": {
      "1337": {
        "on": true,
        "online": true,
        "brightness": 75
      }
    }
  }
}

Initially, all my other tests pass, but the "OnlineOffline" test fails with the following specific error messages. It seems to be having trouble recognizing the device's online status before the test even begins.

FAIL: An OnlineOffline test was not properly handled. Please refer to the test details and re-run.

Online/Offline failed: Device is not online before the test. Please make sure the device online.: expected false to be true

Online/Offline failed: Device state is not changed after 5 minutes.: expected false to be true

The error messages suggest the Test Suite is not seeing the device as online initially, and therefore cannot test the state changes. This is unexpected, as my QUERY response clearly indicates online: true. Is there a specific behavior of the Test Suite that could be causing it to fail to read the initial online status of the device, especially with "willReportState" set to false?

Any guidance on how to resolve this specific set of errors would be greatly appreciated.

Thank you in advance for your help!

[arm_dpe · Accepted solution]

Thanks for sharing. The Test Suite is failing because it's not receiving a definitive "online" signal before the test starts. By setting willReportState: true and using the REPORT_STATE intent, you directly push the device's online status to Google. This eliminates any timing issues or race conditions that can occur with a polling-only approach. We highly recommend updating your implementation to use willReportState: true.

This is the most reliable and recommended way to pass the "OnlineOffline" test.