Fleet 4.73.0 QA: Ensuring A Stable Release

by Marta Kowalska 43 views

Hey guys! Today, we're diving deep into the QA process for Fleet 4.73.0. Our main goal here is to make sure this release is rock-solid and super stable. This article provides easy-to-follow test steps for manually checking the release, so let’s jump right in!

Important Reference Data

Before we get started, here's some crucial info you'll need:

  1. fleetctl preview setup
  2. Permissions documentation
  3. Premium tests require a license key (needs renewal): fleetctl preview --license-key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJGbGVldCBEZXZpY2UgTWFuYWdlbWVudCBJbmMuIiwiZXhwIjoxNjQwOTk1MjAwLCJzdWIiOiJkZXZlbG9wbWVudCIsImRldmljZXMiOjEwMCwibm90ZSI6ImZvciBkZXZlbG9wbWVudCBvbmx5IiwidGllciI6ImJhc2ljIiwiaWF0IjoxNjIyNDI2NTg2fQ.WmZ0kG4seW3IrNvULCHUPBSfFdqj38A_eiXdV_DFunMHechjHbkwtfkf1J6JQJoDyqn8raXpgbdhafDwv3rmDw
  4. Premium tests require an active license key (Expires Sunday, January 1, 2023 12:00:00 AM): fleetctl preview --license-key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJGbGVldCBEZXZpY2UgTWFuYWdlbWVudCBJbmMuIiwiZXhwIjoxNjcyNTMxMjAwLCJzdWIiOiJGbGVldCBEZXZpY2UgTWFuYWdlbWVudCIsImRldmljZXMiOjEwMCwibm90ZSI6ImZvciBkZXZlbG9wbWVudCBvbmx5IiwidGllciI6InByZW1pdW0iLCJpYXQiOjE2NDI1MjIxODF9.EGHQjIzM73YyMbnCruswzg360DEYCsDi9uz48YcDwQHq90BabGT5PIXRiculw79emGj5sk2aKgccTd2hU5J7Jw

Smoke Tests

Smoke tests are crucial. They focus on core functionality and act as the final pre-release review. If these tests fail, the release is a no-go. So, pay close attention!

Fleet Core

Before we start, make sure you have the Fleet version handy (check the "My account" page in the Fleet UI or run fleetctl version) and note your Web browser (e.g., Chrome 88.0.4324).

Prerequisites

  1. fleetctl preview is set up and running the desired test version using --tag parameters. This ensures we're testing the correct build.
  2. Unless you're testing older browser versions specifically, make sure your browser is up to date. Outdated browsers can cause unexpected issues.
  3. Certificates and flag files are in place to create a new host. This is essential for proper host enrollment.
  4. In your browser, clear local storage using devtools. This gives us a clean testing environment.

Orchestration

Let's walk through some key test scenarios:

Test name Step instructions Expected result pass/fail
Update flow 1. Remove all Fleet processes/agents/etc. using fleetctl preview reset for a clean slate.
2. Run fleetctl preview with no tag for the latest stable version.
3. Create a host/query to later confirm the upgrade.
4. STOP fleet-preview-server instances in containers/apps on Docker.
5. Run fleetctl preview with the appropriate testing tag. All previously created hosts/queries are verified to still exist. This ensures no data loss during updates. pass/fail
Login flow 1. Navigate to the login page and attempt to log in with both valid and invalid credentials to verify expected results.
2. Navigate to the login page and attempt to log in with both valid and invalid SSO credentials to verify expected results. 1. Text fields prompt when blank.
2. The correct error message is "authentication failed".
3. The "Forgot password" link prompts for an email.
4. Valid credentials result in a successful login.
5. Valid SSO credentials result in a successful login. pass/fail
Packs flow Verify the management, operation, and logging of "2017 packs". These are crucial for query management. 1. Packs successfully run on host machines after migrations.
2. New Packs can be created.
3. Packs can be edited and deleted.
4. Packs results information is logged. pass/fail
Log destination flow Verify the log destination for software, query, policy, and packs. This ensures logs are being captured correctly. 1. Software, query, policy, and packs logs are successfully sent to external log destinations.
2. Software, query, policy, and packs logs are successfully sent to Filesystem log destinations. pass/fail
OS settings Verify OS settings functionality. 1. Verify the ability to configure Disk encryption (macOS, Windows, & Linux).
2. Verify a host enrolled with Disk encryption enforced successfully encrypts. pass/fail

MDM (Mobile Device Management)

Moving on to MDM, let's ensure all device management features are working smoothly. This section is super important for anyone managing mobile devices with Fleet.

Test name Step instructions Expected result pass/fail
MDM enrollment flow Verify MDM enrollments and run MDM commands. 1. Erase an ADE-eligible macOS host and verify the ability to complete the automated enrollment flow.
2. With Windows MDM turned On, enroll a Windows host and verify MDM is turned On for the host.
3. Verify the ability to run MDM commands on both macOS and Windows hosts from the CLI. pass/fail
MDM migration flow Verify MDM migration for ADE and non-ADE hosts. This is critical for users upgrading from other MDM solutions. 1. Turn off MDM on an ADE-eligible macOS host and verify that the native, "Device Enrollment" macOS notification appears.
2. On the My device page, follow the "Turn on MDM" instructions and verify that MDM is turned on.
3. Turn off MDM on a non-ADE-eligible macOS host.
4. On the My device page, follow the "Turn on MDM" instructions and verify that MDM is turned on.
5. Verify Windows host migrates from 3rd party MDM to Fleet when automatic migration is turned on. pass/fail
OS settings Verify OS settings functionality. 1. Verify Profiles upload/download/delete (macOS & Windows).
2. Verify Profiles are delivered to the host and applied. pass/fail
Setup experience Verify the macOS Setup experience. This ensures a smooth initial setup for macOS devices. 1. Configure End-user authentication.
2. Upload a Bootstrap package.
3. Add software (FMA, VPP, & Custom pkg).
4. Add a script.
5. Enroll an ADE-eligible macOS host and verify successful authentication.
6. Verify the Bootstrap package is delivered.
7. Verify the SwiftDialogue window displays.
8. Verify software installs and the script runs. pass/fail
OS updates Verify OS updates flow. This is essential for keeping devices secure and up-to-date. 1. Configure OS updates (macOS & Windows).
2. Verify on-device that the Nudge prompt appears (macOS 13).
3. Verify enforcing minimumOS occurs during enrollment (macOS 14+). pass/fail
iOS/iPadOS Verify enrollment, profiles, & software installs. This ensures mobile devices are properly managed. 1. Verify ADE enrollment.
2. Verify OTA enrollment.
3. Verify Profiles are delivered to the host and applied.
4. Verify VPP apps install & display correctly in the Activity feed.
5. Verify Turn Off MDM for BYOD & ADE hosts. pass/fail
Lock & Wipe Verify hosts can be locked & wiped. These are critical security features. 1. Verify locking a host from the Fleet UI (macOS, Windows, & Linux).
2. Verify unlocking a host from the Fleet UI (macOS, Windows, & Linux).
3. Verify wiping a host from the Fleet UI (macOS, Windows, & Linux).
4. Verify wiping and locking hosts using fleetctl (macOS, Windows, & Linux). pass/fail
Certificates Upload APNs cert and ABM token renewal workflow. This ensures seamless device enrollment and management. 1. Renew APNs Certificate.
2. Renew ABM Token.
3. Ensure ADE hosts can enroll. pass/fail

Software

Now, let’s move on to software management. These tests ensure software deployment and management are working as expected.

Test name Step instructions Expected result pass/fail
Query flow Create, edit, run, and delete queries. This is fundamental for data retrieval and analysis. 1. Permissions regarding creating/editing/deleting queries are up to date with documentation.
2. Syntax errors result in error messaging.
3. Queries can be run manually. pass/fail
Host Flow Verify a new host can be added and removed following modal instructions using your own device. This ensures basic host management. 1. Host is added via command line.
2. Host serial number and date added are accurate.
3. Host is not visible after it is deleted.
4. Warning and informational modals show when expected and make sense. pass/fail
My device page Verify the end user's my device page loads successfully. This is crucial for user self-service. 1. Clicking the Fleet desktop item, then "My device" successfully loads the my device page.
2. The "My device" page is populated correctly and as expected.
3. Styling and padding appear correct. pass/fail
Scripts Verify script library and execution. This ensures automation and scripting capabilities are working. 1. Verify able to run a script on all host types from the CLI.
2. Verify scripts library upload/download/delete.
3. From Host details (macOS, Windows, & Linux), run a script that should PASS, verify.
4. From Host details (macOS, Windows, & Linux), run a script that should FAIL, verify.
5. Verify UI loading state and statuses for scripts.
6. Disable scripts globally and verify the inability to run them.
7. Verify scripts display correctly in the Activity feed. pass/fail
Software Verify software library and install/download. This ensures software deployment is working as expected. 1. Verify software library upload/download/delete.
2. From Host details (macOS, Windows, & Linux), run an install that should PASS, verify.
3. From My Device (macOS, Windows, & Linux), the software tab should have self-service items available, verify.
4. Verify UI loading state and statuses for installing software.
5. Verify software installs display correctly in the Activity feed. pass/fail
Migration Test Verify Fleet can migrate to the next version with no issues. This is critical for upgrades. 1. Using the GitHub action https://github.com/fleetdm/fleet/actions/workflows/db-upgrade-test.yml,
2. Using the most recent stable version of Fleet and main, click Run workflow.
3. Enter the Docker tag of the Fleet starting version, e.g., 'v4.64.2'.
4. Enter the Docker tag of the Fleet version to upgrade to, e.g., 'rc-minor-fleet-v4.65.0'.
5. Click Run workflow.
6. The action should complete successfully. pass/fail

All Product Groups

This section covers tests that apply across all product groups. Let’s make sure everything is in sync!

Test name Step instructions Expected result pass/fail
Release blockers Verify there are no outstanding release blocking tickets. This is super important for a smooth release. 1. Check this filter to view all open ~release blocker tickets.
2. If any are found, raise an alarm in the #help-engineering and #g-mdm (or #g-endpoint-ops) channels. pass/fail
Load tests Load tests - minor releases only unless otherwise specified. Verify all load test metrics are within acceptable range on the final build of RC. This ensures stability under load. 1. Check this Google doc to review load test key metrics and checks.
2. After all expected changes have been merged to the RC branch, two load tests will need to be run - a new instance with no data, and a migrated instance.
3. For the new instance with no data, set up a load test environment using the RC branch and allow it at least 24 hours of run time.
4. For the migrated instance, set up a load test environment on the previous minor release branch. Once the environment has been set up and stabilized, follow the instructions in Deploying code changes to fleet to migrate to the RC branch. Monitor the metrics post-migration to determine if any performance issues arise.
5. Record metrics in this spreadsheet for the two load test runs. pass/fail

Notes

Make sure to include any important observations here!

  • Issues found new to this version:
  • Issues found that reproduce in the last stable version:
  • What has not been tested:
  • Include any notes on whether issues should block release or not as needed:


fleetd Agent

Let's shift our focus to the fleetd agent. This section ensures the agent is running smoothly and updating correctly.

Includes updates to:

  • Orbit: True / False
  • Desktop: True / False
  • Chrome extension: True / False

List version changes for any component updates below:

  • Orbit v1.xx.x > v1.xx.x
  • Desktop v1.xx.x > v1.xx.x
  • Chrome extension v1.xx.x > v1.xx.x

Testing Gates for New fleetd Release

The goal here is to ensure the new fleetd is tested and promoted from local > edge > stable channels. This helps us catch any issues early on.

  1. Build a new fleetd from the release candidate branch as needed for Orbit, Desktop, and Chrome Extension.
Test name Step instructions Expected result pass/fail
fleetd local testing 1. Following Testing TUF instructions, create binaries for Mac, Windows, and Ubuntu using your local TUF repository and install them on macOS, Linux, and Windows hosts. This ensures the agent can be installed locally. 1. Confirm the hosts install with the updated version and are working correctly. pass/fail
2. Confirm any new features and/or bug fixes associated with this release are working as intended.
fleetd auto-update tests 1. Conduct the fleetd auto-update n+1 test. This tests the auto-update mechanism.
2. QA certifies the new release by commenting in the issue. 1. The agent successfully auto-updates. pass/fail
2. The issue is certified by QA.
fleetd tests 1. Set up a host in your instance to receive updates from the edge channels. This allows us to test the agent in a pre-release environment.
2. Work with the engineer leading the release to push changes to the edge channel. 1. Confirm the hosts running on the edge channel receive the update and are working correctly. pass/fail
2. Confirm any new features and/or bug fixes associated with this release are working as intended.

New fleetd Pushed to Edge

Now, let's ensure the fleetd version pushed to the edge channel is working seamlessly with the current released version of Fleet. This step is crucial for maintaining compatibility.

  1. Fleet server is running the latest released version available on the Fleet Releases page.

  2. Set Agent options to use edge in the Fleet server configuration. For example:

    update_channels:

    osqueryd: edge

    orbit: edge

    desktop: edge

Test name Step instructions Expected result pass/fail
Query flow Run queries. This ensures the agent can execute queries correctly. 1. Queries can be run manually. pass/fail
Host Flow Verify a new host can be added using your own device. This ensures basic host enrollment is functioning. 1. Hosts can enroll and report the correct version of fleetd (orbit, osquery, desktop).
2. Refetching host vitals completes and returns updated information. pass/fail
My device page Verify the end user's my device page loads successfully. This is crucial for user self-service. 1. Clicking the Fleet desktop item, then "My device" successfully loads the my device page.
2. The "My device" page is populated correctly and as expected.
3. Styling and padding appear correct. pass/fail
Scripts Verify script execution. This ensures automation and scripting capabilities are working. 1. Verify able to run a script on all host types from the CLI.
2. From Host details (macOS, Windows, & Linux), run a script that should PASS, verify.
3. From Host details (macOS, Windows, & Linux), run a script that should FAIL, verify.
4. Verify script results display correctly in the Activity feed. pass/fail
Software Verify software install/download. This ensures software deployment is functioning. 1. From Host details (macOS, Windows, & Linux), run an install that should PASS, verify.
2. From My Device (macOS, Windows, & Linux), the software tab should have self-service items available, verify.
3. Verify software installs display correctly in the Activity feed. pass/fail
OS settings Verify OS settings functionality. This ensures OS-level configurations are working. 1. Verify able to configure Disk encryption (macOS, Windows, & Linux).
2. Verify a host enrolled with Disk encryption enforced successfully encrypts. pass/fail
Packs flow Verify the management, operation, and logging of "2017 packs". This ensures query management is working. 1. Packs successfully run on host machines after migrations.
2. New Packs can be created.
3. Packs can be edited and deleted.
4. Packs results information is logged. pass/fail

Notes

Again, make sure to document any issues or observations here! This helps the team address any problems effectively.

  • Issues found new to this version:
  • Issues found that reproduce in the last stable version:
  • What has not been tested:
  • Include any notes on whether issues should block release or not as needed: