weaviate/weaviate-chaos-engineering
Watch
1
0
Fork
You've already forked weaviate-chaos-engineering
 0
mirror of https://github.com/weaviate/weaviate-chaos-engineering.git synced 2026-11-06 10:07:39 +00:00
Code Issues Projects Releases Packages Wiki Activity
Chaos-Engineering-Style CI Pipelines to make sure Weaviate handles whatever the real world throws at it.
940 commits 308 branches 0 tags 14 MiB
  • Python 51.5%
  • Go 32.3%
  • Shell 15.2%
  • Dockerfile 0.8%
  • TypeScript 0.2%
Find a file
Exact
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Mohamed Badawi 30def07ce7
 fix(mt-latency): allow query p99 up to 239ms with a warning above 139ms (#466)
2026-11-05 15:53:09 +01:00
.github fix(mt-latency): allow query p99 up to 239ms with a warning above 139ms (#466) 2026-11-05 15:53:09 +01:00
apps fix(mt-latency): allow query p99 up to 239ms with a warning above 139ms (#466) 2026-11-05 15:53:09 +01:00
.gitignore Add rolling/single-pod restarts QPS/latency pipeline. 2026-03-12 10:41:21 +01:00
.gitmodules add concurrent multi tenancy pipeline 2023-06-30 13:16:09 +02:00
.pre-commit-config.yaml Set black version to 26.1.0 and reformat files 2026-01-20 11:38:03 +01:00
ann_benchmark.sh fix(ann-benchmark): kill Weaviate in the chaos loop 2026-10-28 12:02:18 +01:00
ann_benchmark_aws.sh Modify all tests to use async indexing 2025-10-17 10:35:49 +02:00
ann_benchmark_gcp.sh Modify all tests to use async indexing 2025-10-17 10:35:49 +02:00
ann_benchmark_quantization.sh fix: disable telemetry on the ann-benchmark VMs instead of shipping a sink 2026-07-23 16:44:59 +02:00
ann_benchmark_quantization_aws.sh Modify all tests to use async indexing 2025-10-17 10:35:49 +02:00
ann_benchmark_quantization_gcp.sh Modify all tests to use async indexing 2025-10-17 10:35:49 +02:00
async_repair.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
backup_and_flush.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
backup_and_restore_crud.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
backup_and_restore_multi_node_crud.sh ci: check telemetry per service so a dropped URL is caught and other PR comments 2026-07-28 13:06:37 +02:00
backup_and_restore_multi_node_out_of_sync.sh ci: check telemetry per service so a dropped URL is caught and other PR comments 2026-07-28 13:06:37 +02:00
backup_and_restore_node_mapping.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
backup_and_restore_version_compatibility.sh ci: check telemetry per service so a dropped URL is caught and other PR comments 2026-07-28 13:06:37 +02:00
batch_import_many_classes.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
batch_insert_mismatch.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
bm25_corruption.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
CLAUDE.md reference a weaviate version that actually exists 2026-02-02 10:48:05 +01:00
cleanup_aws_resources.sh chore: use sudo rm -rf to avoid errors on clean up (#385) 2026-01-12 13:46:44 +01:00
cleanup_cancelled_aws_resources.sh Clean ANN resources in CI (#314) 2025-06-13 11:20:27 +02:00
cleanup_cancelled_gcp_instances.sh Clean ANN resources in CI (#314) 2025-06-13 11:20:27 +02:00
cleanup_gcp_resources.sh Clean ANN resources in CI (#314) 2025-06-13 11:20:27 +02:00
common.sh Capture crash headers in diagnostics and forward DISABLE_RECOVERY_ON_PANIC 2026-08-31 16:26:05 +02:00
compaction_and_cleanup.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
compaction_optimization.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
compaction_roaringset.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
compare_recall_after_restart.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
compare_rest_graphql_while_crashing.sh fix: close the telemetry leak the fail-safe default exposed 2026-07-28 15:03:14 +02:00
concurrent_inverted_index_read_write.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
condensing_while_crash.sh Fix compactv2 version gate to match 1.39+ as intended 2026-07-08 10:32:16 -07:00
consecutive_create_and_update_operations.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
counting_while_compacting.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
debug-reindexing.sh ci: check telemetry per service so a dropped URL is caught and other PR comments 2026-07-28 13:06:37 +02:00
delete_and_recreate_class.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
deletes_with_node_out_of_sync.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
downgrade_journey_raft_schema.sh chore: use sudo rm -rf to avoid errors on clean up (#385) 2026-01-12 13:46:44 +01:00
filter_memory_leak.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
geo_crash.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
get_env_var.py Modify all tests to use async indexing 2025-10-17 10:35:49 +02:00
goroutine_leak_on_class_delete.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
hnsw_snapshotting_while_crash.sh Fix compactv2 version gate to match 1.39+ as intended 2026-07-08 10:32:16 -07:00
import_while_crashing.sh fix: close the telemetry leak the fail-safe default exposed 2026-07-28 15:03:14 +02:00
import_while_crashing_no_vector.sh fix: close the telemetry leak the fail-safe default exposed 2026-07-28 15:03:14 +02:00
install_docker_ubuntu.sh fix(install): harden Docker install on Ubuntu 22.04 EC2 2025-09-29 14:43:56 +02:00
LICENSE Bump year 2022-01-01 16:05:07 +01:00
multi_node_ref_imports.sh Merge pull request #395 from weaviate/static-ip-addresses-in-clusters 2026-02-02 11:11:51 +01:00
multi_tenancy_activate_deactivate.sh Merge pull request #395 from weaviate/static-ip-addresses-in-clusters 2026-02-02 11:11:51 +01:00
multi_tenancy_activate_deactivate_v1_24.sh Merge pull request #395 from weaviate/static-ip-addresses-in-clusters 2026-02-02 11:11:51 +01:00
multi_tenancy_concurrent_importing.sh Merge pull request #395 from weaviate/static-ip-addresses-in-clusters 2026-02-02 11:11:51 +01:00
namespace_graduation.sh test(namespace-graduation): catch RBAC loss when a namespace has only built-in roles 2026-10-08 10:44:02 +01:00
pyproject.toml add black linter 2023-02-21 12:56:59 +01:00
README.md doc(README): Update how to run the Github action workflow 2025-01-28 14:04:11 +01:00
replication_importing_while_crashing.sh fix: close the telemetry leak the fail-safe default exposed 2026-07-28 15:03:14 +02:00
replication_importing_with_backup.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
replication_tunable_consistency.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
rest_patch_stops_working_after_restart.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
segfault_batch_ref.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
segfault_filtered_vector_search.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
server_side_batching_with_rolling_restart.sh Improve logging of test 2026-02-20 15:28:51 +00:00
tombstones_cleanup_while_crash.sh Add service names to wait_weaviate for better failure diagnostics (#394) 2026-02-02 08:54:28 +01:00
update_stability.sh fix: disable telemetry on the ann-benchmark VMs instead of shipping a sink 2026-07-23 16:44:59 +02:00
upgrade_journey.sh chore: use sudo rm -rf to avoid errors on clean up (#385) 2026-01-12 13:46:44 +01:00
upgrade_journey_object_nested_properties.sh Remove Scarf's proxy reference. 2025-03-10 07:25:11 +01:00
upgrade_single_node_journey.sh ci: check telemetry per service so a dropped URL is caught and other PR comments 2026-07-28 13:06:37 +02:00

README.md

weaviate-chaos-engineering Weaviate logo

Chaos-Engineering-Style pipelines to make sure Weaviate behaves correctly, even when we try to sabotage it a bit.

What this does

Currently there are two independent pipelines:

Pipeline 1: Importing while Crashing

This pipelines is a Chaos-Monkey-style pipeline where we aim to import a large dataset, but an external source constantly introduces crashes. The goal is to make sure that every crash is recoverable and after the importer is through all objects were indeed imported.

Pipeline 2: Compare Recall after restarts

This pipelines imports a specific dataset (which will be generated as part of the CI) and meassures the recall on it. Then Weaviate is stopped and restarted with all state coming from disk and the recall is measured again. This pipeline makes sure that all async operations we perform on commit logs (combining, condensing, etc.) do not degrade the quality of the vector index.

How to Run

Requirements

  • Bash
  • Docker / Docker Compose
  • jq

Run the scripts

Everything is entirely containerized and all pipelines can be started with simple bash scripts. You can find the scripts in the root folder, such as ./import_while_crashing.sh and ./compare_recall_after_restart.sh. Or simply check the Github actions YAML files for examples.

Run the Github action workflow

You can also run the chaos pipeline for your public weaviate image. Follow the steps below.

NOTE: You should have right permission to trigger Github action workflow. Usually available for Weaviate core developers

  1. Go to Test Matrix workflow
  2. Click on "Run workflow"
  3. Choose main branch
  4. Enter the public docker image of your weaviate. (e.g: semitechnologies/weaviate:1.25.29-6b28b2813)
  5. Start and observe the workflow

Links