Anomaly detection 

    Administrator setup manual

    1. Enable detail billing export


    First step is to export your usage data to BigQuerry. If you have this feature already enabled, you can skip this step.

    1. In your GCP console navigate into Billing > Billing Export > BigQuery Export tab
    2. Find “Detailed usage cost block” and continue to Edit settings
      anomaly detection manual 1
    3. Select the project, in which you want to collect the detailed usage data. This project can be a new one or an existing one. Similar way, you can select the dataset in which the data will be saved. If you are not utilizing Detail usage cost elsewhere, we recommend you set a new project and dataset for this.

    4. Finish the settings and save. From this point, Google will start saving your usage data daily into the selected BigQuery dataset.


    2. Connect anomaly detection

    Once you have your billing export ready, you can connect our application with your dataset via the service account.

    1. Log in to Anomaly detection at https://anomalies.revolgy.com
      • Please contact us first to grant you access to our tool
    2. Navigate to “New setup” on top of the app
    3. The app will generate a Service account for you in the format of a long email address - you will need to add this service account to have access to the Detail usage data export which we created in Chapter 1
    4. In your Google console navigate to IAM & Admin > IAM > Permissions tab. Be sure you selected the correct project which contains the sage data in the top dropdown menu.
    5. Click on the grant access button
    6. As a participant, add the email address of the service account our App generated in Step 3
    7. Assign roles “BigQuery - Data Editor” and “BigQuery - Job User” to the given Service account. Save changes
    8. In Anomaly detection fill in the human-readable name of the Group of resources you want to add and then copy and paste the project id of the Google console project we created, datasetID we created, and tableId which contains the usage export. You can find all of this when you open the BiqQuery in Google Console.
    9. In case you want to send the newly found anomalies notification anywhere like Zappier or Make, you can add your webhook URL as well. This step can be done later as well.
    10. Once you have everything filled, you can Test the connection and save it.

    At this point, our app will run, generate the Machine learning model of your data monthly, and compare the daily data with it. Anomalies found will be sent to the app and a webhook will be triggered for each newly found anomaly.

    Please note, that at the beginning of the usage of our app due to a lack of historical data, the model is not able to check anomalies correctly. This usually gets better each week with maximum precision after a month or so.

    3. Configure anomaly detection

    You can configure how sensitive should the detection be.

    1. In-app navigate to the Group of resources you are tracking and click on the configure icon
      anomaly detection manual 2
    2. You have a few options to configure:
      Min Difference - Set what will be the minimal daily difference the app should consider as an anomaly
      Threshold (from 0 to 1, default 0.95) - Our model works with the calculation of probability, that the daily data are anomalies - by increasing the number, the app will be less sensitive, by lowering this number app will be more sensitive, and send more anomalies to check.
      anomaly detection manual 3
    3. In case you want to send the newly found anomalies notification anywhere like Zappier or Make, you can add or edit your webhook URL here. The format of the webhook is the following:
      {
        "usageTime": "2023-11-15T00:00:00.000Z",
        "description": "uyaXSBfKTGwosiWSCHUG6ezAOjXQ9bRmC6CXU6vfA8k=|Cloud SQL for MySQL: Regional - vCPU in Americas|u+DYgIFLF8/DBWzZ04uiBdUgC3n3gfyjxzUQHEm9/Rk=|XjWsqMDak+cGzBwRQ3whd9WsClrj4e3rHiUIKI4XgZo=|2C7C-7705-6CE7",
        "difference": 40647420.285667,
        "payload": {
          "projectId": "u+DYgIFLF8/DBWzZ04uiBdUgC3n3gfyjxzUQHEm9/Rk=",
          "projectName": "XjWsqMDak+cGzBwRQ3whd9WsClrj4e3rHiUIKI4XgZo=",
          "skuId": "2C7C-7705-6CE7",
          "billingAccountId": "uyaXSBfKTGwosiWSCHUG6ezAOjXQ9bRmC6CXU6vfA8k=",
          "skuDescription": "Cloud SQL for MySQL: Regional - vCPU in Americas",
          "labels": {
            "firebase": "enabled"
          },
          "diffRelative": 3566621.967645492
        }
      }
    4. In case you want to share your configuration with other people, you can add their email address in the Access section of the configuration