Backend Database Work
Scheduled Maintenance Report for F5 App Infrastructure Protection
Postmortem

On June 30, 2022, the F5 Advanced Infrastructure Protection (Threat Stack) engineering team conducted maintenance which resulted in changes that may have affected your organization temporarily. At this point in time, all issues have been remediated and no outstanding changes regarding these impacts remain. More information about impact and remediation is available below.

Who Was Impacted

  • Customers leveraging File Integrity Monitoring rules, specifically those who may have many FIM rules
  • Customers conducting rule suppression changes during the maintenance window

What Was Impacted

Increased File Integrity Monitoring (FIM) Alerts

A platform change was applied which increased the number of file integrity monitoring rule alerts. As a result, customers may have experienced a temporary increase in alerts due to the broader scope of FIM events being processed. The changes to alerting on FIM events occurred in the platform and not on the host level (ie. no agent changes were made). The additional data being processed through the rules engine was already being collected by the agent and sent to the platform prior to the platform changes. This change has no impact on agent performance.

UI/API Suppression Updates

The platform change involved updates to our internal suppressions API to enable new future functionality. During the maintenance window, modifications to rule suppressions could result in the deletion of other suppressions for that specific rule.

When Was The Impact

Start: 9 AM EST

End:

  • Increased FIM Alerts: 7:10 PM EST
  • UI/API Suppression Updates: 8:04 PM EST

Impact Remediation

Increased File Integrity Monitoring (FIM) Alerts

The F5 Advanced Infrastructure Protection team added a suppression to the relevant FIM rules. The suppression applied to those FIM rules is command != null AND filename != null. This suppression restored the level of alerting to the state in which it was prior to the data platform maintenance.

As a result of this new functionality, this suppression can be removed for troubleshooting FIM events when a broader number of inotify events need to be collected for investigation.

UI/API Suppression Updates

The F5 Advanced Infrastructure Protection team restored suppressions for affected rules to their original state. The implementation of suppressions subject to this was completed at approximately 10 AM EST on Friday July 1, 2022.

Questions? Concerns?

We’re here to help. Please reach out to support@threatstack.com with any further assistance with any questions or concerns.

Posted Jul 01, 2022 - 15:40 EDT

Completed
The scheduled maintenance has been completed.
Posted Jun 30, 2022 - 09:30 EDT
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Jun 30, 2022 - 09:00 EDT
Scheduled
We will be performing maintenance to backend services that power agent registration, authentication, and rule assignment. During this time new agents will not be able to register with or connect to the Threat Stack platform. Connected agents will continue to operate as expected.

While this maintenance is necessary to power the future of rules management in the platform, we do apologize for the inconvenience.
Posted Jun 22, 2022 - 11:01 EDT
This scheduled maintenance affected: Web UI, API, Cloudtrail Integration and EC2 Sync, and S3 Data Portability.