Missing data EmonCMS


#1

Hi,

My emoncms service was unavailable for a period of time, from about 1am - 5am today. I restarted the services and emoncms was restored, but the iotawatt unit did not update the data for the period that emoncms was unavailable.

I’ve check that the data is available on iotawatt. What can I do to
1.) backfill the gaps in emoncms and
2.) ensure it does not happen in future.

Any ideas?

Message log from iotawatt:
10/23/18 01:03:22 EmonService: HTTP response -4, retrying.
10/23/18 05:03:02 EmonService: Retry successful after 522 attempts.

Graph from iotawatt:

Graph from emoncms for the same input:

Data from iotawatt for this period:
2018-10-23 01:02:00, 547.8
2018-10-23 01:03:00, 477.8
2018-10-23 01:04:00, 477.4
2018-10-23 01:05:00, 478.8
2018-10-23 01:06:00, 480.6
2018-10-23 01:07:00, 473.7
2018-10-23 01:08:00, 478.3
2018-10-23 01:09:00, 483.8
2018-10-23 01:10:00, 405.7
2018-10-23 01:11:00, 375.1
2018-10-23 01:12:00, 374.7
2018-10-23 01:13:00, 377.3
2018-10-23 01:14:00, 380.5
2018-10-23 01:15:00, 373.7
2018-10-23 01:16:00, 372.5
2018-10-23 01:17:00, 366.9
2018-10-23 01:18:00, 363.8
2018-10-23 01:19:00, 363.3
2018-10-23 01:20:00, 363.2
2018-10-23 01:21:00, 364.1
2018-10-23 01:22:00, 362.8
2018-10-23 01:23:00, 363.1
2018-10-23 01:24:00, 420.7
2018-10-23 01:25:00, 472.9
2018-10-23 01:26:00, 541.6
2018-10-23 01:27:00, 555.4
2018-10-23 01:28:00, 551.3
2018-10-23 01:29:00, 550.8
2018-10-23 01:30:00, 551.6
2018-10-23 01:31:00, 544.4
2018-10-23 01:32:00, 532.1
2018-10-23 01:33:00, 531.9
2018-10-23 01:34:00, 531.2
2018-10-23 01:35:00, 528.8
2018-10-23 01:36:00, 528.5
2018-10-23 01:37:00, 526.9
2018-10-23 01:38:00, 526.1
2018-10-23 01:39:00, 526.0
2018-10-23 01:40:00, 525.1
2018-10-23 01:41:00, 526.5
2018-10-23 01:42:00, 488.0
2018-10-23 01:43:00, 460.0
2018-10-23 01:44:00, 461.2
2018-10-23 01:45:00, 461.3
2018-10-23 01:46:00, 369.8
2018-10-23 01:47:00, 362.2
2018-10-23 01:48:00, 362.2
2018-10-23 01:49:00, 362.5
2018-10-23 01:50:00, 363.2
2018-10-23 01:51:00, 362.6
2018-10-23 01:52:00, 361.8
2018-10-23 01:53:00, 356.1
2018-10-23 01:54:00, 355.3
2018-10-23 01:55:00, 356.4
2018-10-23 01:56:00, 356.6
2018-10-23 01:57:00, 356.8
2018-10-23 01:58:00, 355.8
2018-10-23 01:59:00, 356.6
2018-10-23 02:00:00, 406.9
2018-10-23 02:01:00, 467.8
2018-10-23 02:02:00, 465.2
2018-10-23 02:03:00, 465.7
2018-10-23 02:04:00, 464.8
2018-10-23 02:05:00, 481.8
2018-10-23 02:06:00, 548.1
2018-10-23 02:07:00, 545.3
2018-10-23 02:08:00, 541.9
2018-10-23 02:09:00, 540.8
2018-10-23 02:10:00, 539.7
2018-10-23 02:11:00, 540.0
2018-10-23 02:12:00, 538.6
2018-10-23 02:13:00, 537.2
2018-10-23 02:14:00, 536.1
2018-10-23 02:15:00, 541.6
2018-10-23 02:16:00, 549.1
2018-10-23 02:17:00, 544.8
2018-10-23 02:18:00, 540.1
2018-10-23 02:19:00, 538.3
2018-10-23 02:20:00, 537.0
2018-10-23 02:21:00, 538.1
2018-10-23 02:22:00, 375.3
2018-10-23 02:23:00, 367.2
2018-10-23 02:24:00, 366.0
2018-10-23 02:25:00, 366.3
2018-10-23 02:26:00, 367.3
2018-10-23 02:27:00, 367.4
2018-10-23 02:28:00, 370.8
2018-10-23 02:29:00, 366.5
2018-10-23 02:30:00, 364.2
2018-10-23 02:31:00, 365.7
2018-10-23 02:32:00, 368.2
2018-10-23 02:33:00, 366.2
2018-10-23 02:34:00, 365.6
2018-10-23 02:35:00, 364.7
2018-10-23 02:36:00, 380.4
2018-10-23 02:37:00, 477.2
2018-10-23 02:38:00, 473.7
2018-10-23 02:39:00, 473.0
2018-10-23 02:40:00, 472.5
2018-10-23 02:41:00, 472.8
2018-10-23 02:42:00, 471.3
2018-10-23 02:43:00, 470.4
2018-10-23 02:44:00, 469.9
2018-10-23 02:45:00, 504.0
2018-10-23 02:46:00, 553.2
2018-10-23 02:47:00, 550.1
2018-10-23 02:48:00, 544.2
2018-10-23 02:49:00, 540.3
2018-10-23 02:50:00, 536.8
2018-10-23 02:51:00, 536.4
2018-10-23 02:52:00, 533.0
2018-10-23 02:53:00, 530.8
2018-10-23 02:54:00, 528.7
2018-10-23 02:55:00, 528.1
2018-10-23 02:56:00, 530.1
2018-10-23 02:57:00, 508.4
2018-10-23 02:58:00, 428.3
2018-10-23 02:59:00, 427.8
2018-10-23 03:00:00, 427.4
2018-10-23 03:01:00, 390.6
2018-10-23 03:02:00, 360.6
2018-10-23 03:03:00, 360.8
2018-10-23 03:04:00, 360.2
2018-10-23 03:05:00, 353.3
2018-10-23 03:06:00, 352.7
2018-10-23 03:07:00, 352.0
2018-10-23 03:08:00, 351.5
2018-10-23 03:09:00, 351.5
2018-10-23 03:10:00, 352.0
2018-10-23 03:11:00, 352.0
2018-10-23 03:12:00, 400.6
2018-10-23 03:13:00, 461.3
2018-10-23 03:14:00, 461.1
2018-10-23 03:15:00, 464.9
2018-10-23 03:16:00, 465.7
2018-10-23 03:17:00, 463.8
2018-10-23 03:18:00, 462.1
2018-10-23 03:19:00, 462.1
2018-10-23 03:20:00, 462.0
2018-10-23 03:21:00, 462.7
2018-10-23 03:22:00, 461.6
2018-10-23 03:23:00, 461.4
2018-10-23 03:24:00, 460.8
2018-10-23 03:25:00, 524.0
2018-10-23 03:26:00, 542.8
2018-10-23 03:27:00, 539.0
2018-10-23 03:28:00, 538.2
2018-10-23 03:29:00, 537.1
2018-10-23 03:30:00, 535.5
2018-10-23 03:31:00, 532.0
2018-10-23 03:32:00, 527.0
2018-10-23 03:33:00, 437.2
2018-10-23 03:34:00, 421.8
2018-10-23 03:35:00, 418.9
2018-10-23 03:36:00, 418.1
2018-10-23 03:37:00, 416.9
2018-10-23 03:38:00, 417.3
2018-10-23 03:39:00, 417.6
2018-10-23 03:40:00, 416.3
2018-10-23 03:41:00, 364.4
2018-10-23 03:42:00, 349.2
2018-10-23 03:43:00, 350.0
2018-10-23 03:44:00, 350.3
2018-10-23 03:45:00, 350.0
2018-10-23 03:46:00, 349.1
2018-10-23 03:47:00, 379.9
2018-10-23 03:48:00, 461.2
2018-10-23 03:49:00, 459.7
2018-10-23 03:50:00, 460.5
2018-10-23 03:51:00, 460.4
2018-10-23 03:52:00, 460.1
2018-10-23 03:53:00, 460.4
2018-10-23 03:54:00, 460.3
2018-10-23 03:55:00, 460.1
2018-10-23 03:56:00, 458.2
2018-10-23 03:57:00, 457.1
2018-10-23 03:58:00, 456.9
2018-10-23 03:59:00, 455.9
2018-10-23 04:00:00, 456.7
2018-10-23 04:01:00, 462.2
2018-10-23 04:02:00, 458.9
2018-10-23 04:03:00, 460.4
2018-10-23 04:04:00, 460.6
2018-10-23 04:05:00, 541.2
2018-10-23 04:06:00, 540.8
2018-10-23 04:07:00, 537.2
2018-10-23 04:08:00, 450.3
2018-10-23 04:09:00, 437.8
2018-10-23 04:10:00, 448.9
2018-10-23 04:11:00, 450.8
2018-10-23 04:12:00, 450.3
2018-10-23 04:13:00, 449.5
2018-10-23 04:14:00, 448.8
2018-10-23 04:15:00, 446.6
2018-10-23 04:16:00, 442.0
2018-10-23 04:17:00, 440.7
2018-10-23 04:18:00, 439.6
2018-10-23 04:19:00, 438.0
2018-10-23 04:20:00, 428.6
2018-10-23 04:21:00, 364.4
2018-10-23 04:22:00, 385.9
2018-10-23 04:23:00, 468.4
2018-10-23 04:24:00, 465.5
2018-10-23 04:25:00, 465.0
2018-10-23 04:26:00, 465.7
2018-10-23 04:27:00, 465.3
2018-10-23 04:28:00, 466.6
2018-10-23 04:29:00, 465.0
2018-10-23 04:30:00, 471.7
2018-10-23 04:31:00, 474.8
2018-10-23 04:32:00, 473.1
2018-10-23 04:33:00, 474.4
2018-10-23 04:34:00, 473.9
2018-10-23 04:35:00, 472.8
2018-10-23 04:36:00, 469.6
2018-10-23 04:37:00, 461.2
2018-10-23 04:38:00, 459.5
2018-10-23 04:39:00, 459.9
2018-10-23 04:40:00, 455.6
2018-10-23 04:41:00, 464.3
2018-10-23 04:42:00, 437.7
2018-10-23 04:43:00, 363.9
2018-10-23 04:44:00, 384.8
2018-10-23 04:45:00, 445.4
2018-10-23 04:46:00, 440.4
2018-10-23 04:47:00, 437.6
2018-10-23 04:48:00, 436.7
2018-10-23 04:49:00, 435.6
2018-10-23 04:50:00, 429.6
2018-10-23 04:51:00, 424.2
2018-10-23 04:52:00, 423.5
2018-10-23 04:53:00, 424.5
2018-10-23 04:54:00, 424.4
2018-10-23 04:55:00, 423.0
2018-10-23 04:56:00, 868.0
2018-10-23 04:57:00, 590.3
2018-10-23 04:58:00, 850.6
2018-10-23 04:59:00, 1247.9
2018-10-23 05:00:00, 631.2
2018-10-23 05:01:00, 480.6
2018-10-23 05:02:00, 481.9
2018-10-23 05:03:00, 477.7

Data from input on emoncms:
2018-10-23 01:02:00, 483.0
2018-10-23 01:03:00, null
2018-10-23 01:04:00, null
2018-10-23 01:05:00, null
2018-10-23 01:06:00, null
2018-10-23 01:07:00, null
2018-10-23 01:08:00, null
2018-10-23 01:09:00, null
2018-10-23 01:10:00, null
2018-10-23 01:11:00, null
2018-10-23 01:12:00, null
2018-10-23 01:13:00, null
2018-10-23 01:14:00, null
2018-10-23 01:15:00, null
2018-10-23 01:16:00, null
2018-10-23 01:17:00, null
2018-10-23 01:18:00, null
2018-10-23 01:19:00, null
2018-10-23 01:20:00, null
2018-10-23 01:21:00, null
2018-10-23 01:22:00, null
2018-10-23 01:23:00, null
2018-10-23 01:24:00, null
2018-10-23 01:25:00, null
2018-10-23 01:26:00, null
2018-10-23 01:27:00, null
2018-10-23 01:28:00, null
2018-10-23 01:29:00, null
2018-10-23 01:30:00, null
2018-10-23 01:31:00, null
2018-10-23 01:32:00, null
2018-10-23 01:33:00, null
2018-10-23 01:34:00, null
2018-10-23 01:35:00, null
2018-10-23 01:36:00, null
2018-10-23 01:37:00, null
2018-10-23 01:38:00, null
2018-10-23 01:39:00, null
2018-10-23 01:40:00, null
2018-10-23 01:41:00, null
2018-10-23 01:42:00, null
2018-10-23 01:43:00, null
2018-10-23 01:44:00, null
2018-10-23 01:45:00, null
2018-10-23 01:46:00, null
2018-10-23 01:47:00, null
2018-10-23 01:48:00, null
2018-10-23 01:49:00, null
2018-10-23 01:50:00, null
2018-10-23 01:51:00, null
2018-10-23 01:52:00, null
2018-10-23 01:53:00, null
2018-10-23 01:54:00, null
2018-10-23 01:55:00, null
2018-10-23 01:56:00, null
2018-10-23 01:57:00, null
2018-10-23 01:58:00, null
2018-10-23 01:59:00, null
2018-10-23 02:00:00, null
2018-10-23 02:01:00, null
2018-10-23 02:02:00, null
2018-10-23 02:03:00, null
2018-10-23 02:04:00, null
2018-10-23 02:05:00, null
2018-10-23 02:06:00, null
2018-10-23 02:07:00, null
2018-10-23 02:08:00, null
2018-10-23 02:09:00, null
2018-10-23 02:10:00, null
2018-10-23 02:11:00, null
2018-10-23 02:12:00, null
2018-10-23 02:13:00, null
2018-10-23 02:14:00, null
2018-10-23 02:15:00, null
2018-10-23 02:16:00, null
2018-10-23 02:17:00, null
2018-10-23 02:18:00, null
2018-10-23 02:19:00, null
2018-10-23 02:20:00, null
2018-10-23 02:21:00, null
2018-10-23 02:22:00, null
2018-10-23 02:23:00, null
2018-10-23 02:24:00, null
2018-10-23 02:25:00, null
2018-10-23 02:26:00, null
2018-10-23 02:27:00, null
2018-10-23 02:28:00, null
2018-10-23 02:29:00, null
2018-10-23 02:30:00, null
2018-10-23 02:31:00, null
2018-10-23 02:32:00, null
2018-10-23 02:33:00, null
2018-10-23 02:34:00, null
2018-10-23 02:35:00, null
2018-10-23 02:36:00, null
2018-10-23 02:37:00, null
2018-10-23 02:38:00, null
2018-10-23 02:39:00, null
2018-10-23 02:40:00, null
2018-10-23 02:41:00, null
2018-10-23 02:42:00, null
2018-10-23 02:43:00, null
2018-10-23 02:44:00, null
2018-10-23 02:45:00, null
2018-10-23 02:46:00, null
2018-10-23 02:47:00, null
2018-10-23 02:48:00, null
2018-10-23 02:49:00, null
2018-10-23 02:50:00, null
2018-10-23 02:51:00, null
2018-10-23 02:52:00, null
2018-10-23 02:53:00, null
2018-10-23 02:54:00, null
2018-10-23 02:55:00, null
2018-10-23 02:56:00, null
2018-10-23 02:57:00, null
2018-10-23 02:58:00, null
2018-10-23 02:59:00, null
2018-10-23 03:00:00, null
2018-10-23 03:01:00, null
2018-10-23 03:02:00, null
2018-10-23 03:03:00, null
2018-10-23 03:04:00, null
2018-10-23 03:05:00, null
2018-10-23 03:06:00, null
2018-10-23 03:07:00, null
2018-10-23 03:08:00, null
2018-10-23 03:09:00, null
2018-10-23 03:10:00, null
2018-10-23 03:11:00, null
2018-10-23 03:12:00, null
2018-10-23 03:13:00, null
2018-10-23 03:14:00, null
2018-10-23 03:15:00, null
2018-10-23 03:16:00, null
2018-10-23 03:17:00, null
2018-10-23 03:18:00, null
2018-10-23 03:19:00, null
2018-10-23 03:20:00, null
2018-10-23 03:21:00, null
2018-10-23 03:22:00, null
2018-10-23 03:23:00, null
2018-10-23 03:24:00, null
2018-10-23 03:25:00, null
2018-10-23 03:26:00, null
2018-10-23 03:27:00, null
2018-10-23 03:28:00, null
2018-10-23 03:29:00, null
2018-10-23 03:30:00, null
2018-10-23 03:31:00, null
2018-10-23 03:32:00, null
2018-10-23 03:33:00, null
2018-10-23 03:34:00, null
2018-10-23 03:35:00, null
2018-10-23 03:36:00, null
2018-10-23 03:37:00, null
2018-10-23 03:38:00, null
2018-10-23 03:39:00, null
2018-10-23 03:40:00, null
2018-10-23 03:41:00, null
2018-10-23 03:42:00, null
2018-10-23 03:43:00, null
2018-10-23 03:44:00, null
2018-10-23 03:45:00, null
2018-10-23 03:46:00, null
2018-10-23 03:47:00, null
2018-10-23 03:48:00, null
2018-10-23 03:49:00, null
2018-10-23 03:50:00, null
2018-10-23 03:51:00, null
2018-10-23 03:52:00, null
2018-10-23 03:53:00, null
2018-10-23 03:54:00, null
2018-10-23 03:55:00, null
2018-10-23 03:56:00, null
2018-10-23 03:57:00, null
2018-10-23 03:58:00, null
2018-10-23 03:59:00, null
2018-10-23 04:00:00, null
2018-10-23 04:01:00, null
2018-10-23 04:02:00, null
2018-10-23 04:03:00, null
2018-10-23 04:04:00, null
2018-10-23 04:05:00, null
2018-10-23 04:06:00, null
2018-10-23 04:07:00, null
2018-10-23 04:08:00, null
2018-10-23 04:09:00, null
2018-10-23 04:10:00, null
2018-10-23 04:11:00, null
2018-10-23 04:12:00, null
2018-10-23 04:13:00, null
2018-10-23 04:14:00, null
2018-10-23 04:15:00, null
2018-10-23 04:16:00, null
2018-10-23 04:17:00, null
2018-10-23 04:18:00, null
2018-10-23 04:19:00, null
2018-10-23 04:20:00, null
2018-10-23 04:21:00, null
2018-10-23 04:22:00, null
2018-10-23 04:23:00, null
2018-10-23 04:24:00, null
2018-10-23 04:25:00, null
2018-10-23 04:26:00, null
2018-10-23 04:27:00, null
2018-10-23 04:28:00, null
2018-10-23 04:29:00, null
2018-10-23 04:30:00, null
2018-10-23 04:31:00, null
2018-10-23 04:32:00, null
2018-10-23 04:33:00, null
2018-10-23 04:34:00, null
2018-10-23 04:35:00, null
2018-10-23 04:36:00, null
2018-10-23 04:37:00, null
2018-10-23 04:38:00, null
2018-10-23 04:39:00, null
2018-10-23 04:40:00, null
2018-10-23 04:41:00, null
2018-10-23 04:42:00, null
2018-10-23 04:43:00, null
2018-10-23 04:44:00, null
2018-10-23 04:45:00, null
2018-10-23 04:46:00, null
2018-10-23 04:47:00, null
2018-10-23 04:48:00, null
2018-10-23 04:49:00, null
2018-10-23 04:50:00, null
2018-10-23 04:51:00, null
2018-10-23 04:52:00, null
2018-10-23 04:53:00, null
2018-10-23 04:54:00, null
2018-10-23 04:55:00, null
2018-10-23 04:56:00, null
2018-10-23 04:57:00, null
2018-10-23 04:58:00, null
2018-10-23 04:59:00, null
2018-10-23 05:00:00, null
2018-10-23 05:01:00, null
2018-10-23 05:02:00, 473.1
2018-10-23 05:03:00, 474.2

Thanks.


#2

There is no way to backfill that data from IoTaWatt. There may be a way to manually update Emoncms but I haven’t been keeping up with recent improvements to that.

This is a rare event, but I have looked at the code and I will investigate a loophole in the recovery that may be responsible.