Skip to content

Commit 5c7a71a

Browse files
rzabarazeshrzabarazesh
committed
Update readme
Signed-off-by: Reza Barazesh <[email protected]>
1 parent 401097a commit 5c7a71a

File tree

2 files changed

+72
-208
lines changed

2 files changed

+72
-208
lines changed

‎buildkite/pipeline_generator/README.md‎

Lines changed: 53 additions & 99 deletions
Original file line numberDiff line numberDiff line change
@@ -24,42 +24,52 @@ python pipeline_generator.py --pipeline_mode amd
2424
```
2525
pipeline_generator/
2626
├── pipeline_generator.py # Main entry point
27-
├── config.py # Configuration
28-
├── build_config.py # Build step configs
29-
├── hardware_config.py # Hardware test configs
27+
├── pipeline_config.py # Configuration
28+
├── docker_build_configs.py # Build step configs
29+
├── hardware_test_configs.py # Hardware test configs
30+
├── pyproject.toml # Ruff & mypy config
3031
│
31-
├── models/ # Data models
32+
├── ci/ # CI-specific logic
33+
│ ├── ci_pipeline.py # Main CI orchestration
34+
│ ├── docker_builds.py # CI Docker builds
35+
│ ├── docker_plugins.py # CI Docker plugin construction
36+
│ ├── test_step_converter.py # Convert tests to Buildkite steps
37+
│ ├── test_filtering.py # Which tests to run
38+
│ ├── manual_trigger_rules.py # Blocking logic
39+
│ ├── amd_tests.py # AMD test group
40+
│ ├── torch_nightly_tests.py # Torch nightly group
41+
│ └── hardware_tests.py # External hardware tests
42+
│
43+
├── fastcheck/ # Fastcheck-specific logic
44+
│ ├── fastcheck_pipeline.py # Main fastcheck orchestration
45+
│ ├── docker_builds.py # Fastcheck Docker builds
46+
│ ├── docker_plugins.py # Fastcheck Docker plugin construction
47+
│ ├── test_step_converter.py # Convert tests to Buildkite steps
48+
│ ├── test_filtering.py # Which tests to run
49+
│ ├── manual_trigger_rules.py # Blocking logic
50+
│ ├── amd_tests.py # AMD test group (Basic Correctness only)
51+
│ └── hardware_tests.py # Hardware tests (TPU, GH200, Intel)
52+
│
53+
├── data_models/ # Pydantic data models
3254
│ ├── test_step.py # Input from test-pipeline.yaml
3355
│ ├── buildkite_step.py # Output for Buildkite
3456
│ └── docker_config.py # Docker/K8s configs
3557
│
36-
├── steps/ # Step generators (organized by type)
37-
│ ├── build_steps.py # Docker image builds
38-
│ ├── test_steps.py # Regular test steps
39-
│ ├── hardware_steps.py # External hardware (Neuron, TPU, Intel, etc.)
40-
│ └── group_steps.py # Special groups (AMD, Torch Nightly)
41-
│
42-
├── transformers/ # Command transformation pipeline
43-
│ ├── base.py # Base transformer interface
58+
├── command_builders/ # Command transformations (CI only)
4459
│ ├── normalizer.py # Flatten & normalize commands
45-
│ ├── test_targeting.py # Intelligent test targeting
46-
│ └── coverage.py # Coverage injection
47-
│
48-
├── selection/ # Test selection logic
49-
│ ├── filtering.py # Should run/skip decisions
50-
│ └── blocking.py # Block step (manual trigger) logic
60+
│ ├── intelligent_test_selection.py # Intelligent test targeting
61+
│ └── coverage_injection.py # Coverage injection
5162
│
52-
├── docker/ # Docker plugin builders
53-
│ └── plugin_builder.py # Builds Docker/K8s plugins
54-
│
55-
├── utils/ # Utilities
56-
│ ├── constants.py # Enums, constants
57-
│ ├── agents.py # Agent queue selection
58-
│ └── commands.py # Command helpers
63+
├── utils/ # Shared utilities & constants
64+
│ ├── constants.py # All constants (build keys, queues, labels, etc.)
65+
│ ├── agent_queues.py # Agent queue selection
66+
│ ├── command_utils.py # Command helpers
67+
│ └── amd_command_builder.py # AMD command formatting
5968
│
6069
└── tests/ # Test suite
6170
├── test_*.py # 125 unit tests
62-
└── test_integration_comprehensive.py # 56 integration scenarios
71+
├── test_integration_comprehensive.py # 56 CI scenarios
72+
└── test_integration_fastcheck.py # 8 fastcheck scenarios
6373
```
6474

6575
## Main Flow
@@ -88,77 +98,24 @@ def generate(self, test_steps):
8898
return steps
8999
```
90100

91-
This structure keeps the high-level flow readable while organizing details into focused modules.
92-
93-
## Command Transformation Pipeline
94-
95-
One key improvement over Jinja is making command transformations explicit. When converting a test step to a Buildkite step, commands go through:
96-
97-
1. **Flatten** - Multi-node commands (list of lists) become single list
98-
2. **Normalize** - Remove backslashes from YAML line continuations
99-
3. **Test Targeting** - If only test files changed, run just those tests
100-
4. **Coverage** - Inject coverage collection if enabled
101-
5. **Join** - Combine into single command string
102-
103-
This happens in `docker/plugin_builder.py::build_docker_command()`. Adding a new transformation is straightforward - just create a new transformer in `transformers/`.
101+
CI and Fastcheck have completely separate implementations with zero shared logic that has mode checks. This makes it easy to modify one without worrying about breaking the other.
104102

105103
## Where to Find Things
106104

107-
Coming from the Jinja template? Here's where logic moved:
108-
109-
**Build steps** (lines 14-179 in Jinja)
110-
Now in: `steps/build_steps.py`
111-
112-
**Test step conversion** (lines 180-550 in Jinja)
113-
Now in: `steps/test_steps.py` and `docker/plugin_builder.py`
114-
115-
**Test selection/blocking** (lines 508-530, 600-621 in Jinja)
116-
Now in: `selection/blocking.py` and `selection/filtering.py`
117-
118-
**Coverage injection** (lines 33-158 in Jinja)
119-
Now in: `transformers/coverage.py`
120-
121-
**Intelligent test targeting** (lines 20-158 in Jinja)
122-
Now in: `transformers/test_targeting.py` and `selection/filtering.py`
105+
**For CI pipeline:**
106+
- Main logic: `ci/ci_pipeline.py`
107+
- Build steps: `ci/docker_builds.py`
108+
- Test filtering: `ci/test_filtering.py` and `ci/manual_trigger_rules.py`
109+
- AMD/Torch nightly: `ci/amd_tests.py`, `ci/torch_nightly_tests.py`
123110

124-
**AMD tests** (lines 662-727 in Jinja)
125-
Now in: `steps/group_steps.py::generate_amd_group()`
111+
**For Fastcheck pipeline:**
112+
- Main logic: `fastcheck/fastcheck_pipeline.py`
113+
- Everything else in `fastcheck/` directory
126114

127-
**Torch Nightly tests** (lines 579-658 in Jinja)
128-
Now in: `steps/group_steps.py::generate_torch_nightly_group()`
129-
130-
**Hardware tests** (lines 729-863 in Jinja)
131-
Now in: `steps/hardware_steps.py`
132-
133-
## Common Tasks
134-
135-
### Adding a new build variant
136-
Edit `steps/build_steps.py`. Follow the pattern of existing build steps.
137-
138-
### Adding command transformation logic
139-
Create a new transformer in `transformers/`:
140-
141-
```python
142-
from .base import CommandTransformer
143-
144-
class MyTransformer(CommandTransformer):
145-
def transform(self, commands, test_step, config):
146-
if should_apply():
147-
return modified_commands
148-
return None # Falls through to next transformer
149-
```
150-
151-
Then use it in `docker/plugin_builder.py::build_docker_command()`.
152-
153-
### Adding a new hardware platform
154-
Add configuration to `hardware_config.py` and generation logic to `steps/hardware_steps.py`.
155-
156-
### Adjusting Docker plugin configuration
157-
Look in `docker/plugin_builder.py` for the plugin builder logic, or `models/docker_config.py` for the config dataclasses.
158-
159-
### Changing test selection rules
160-
- Run/skip decisions: `selection/filtering.py`
161-
- Block (manual trigger) decisions: `selection/blocking.py`
115+
**Shared code:**
116+
- Constants: `utils/constants.py` (build keys, queues, labels, etc.)
117+
- Data models: `data_models/`
118+
- Config files: `*_config.py` at root
162119

163120
## Testing
164121

@@ -169,14 +126,11 @@ python -m pytest tests/ -k "not integration" -v
169126

170127
Run integration tests (verifies 100% compatibility with Jinja):
171128
```bash
172-
# CI mode (56 scenarios)
173-
python tests/test_integration_comprehensive.py
174-
175-
# Fastcheck mode (8 scenarios)
176-
python tests/test_integration_fastcheck.py
129+
# All integration tests via pytest
130+
pytest tests/test_integration_comprehensive.py tests/test_integration_fastcheck.py
177131

178-
# AMD mode (not yet implemented)
179-
python tests/test_integration_amd.py
132+
# Or run specific scenario
133+
pytest tests/test_integration_comprehensive.py -k "coverage"
180134
```
181135

182136
**Status**: CI and Fastcheck modes achieve 100% YAML compatibility with their respective Jinja templates.

‎buildkite/pipeline_generator/tests/README.md‎

Lines changed: 19 additions & 109 deletions
Original file line numberDiff line numberDiff line change
@@ -12,123 +12,33 @@ Comprehensive backward compatibility tests verifying 100% YAML equivalence with
1212

1313
## Running Tests
1414

15-
### Run all unit tests
1615
```bash
17-
cd /Users/rezabarazesh/Documents/test/ci-infra/buildkite/pipeline_generator
18-
python -m pytest tests/ -v -k "not integration"
19-
```
20-
21-
### Run integration tests (comprehensive)
22-
```bash
23-
cd /Users/rezabarazesh/Documents/test/ci-infra/buildkite/pipeline_generator/tests
24-
python test_integration_comprehensive.py
25-
# or
26-
./test_integration_comprehensive.py
27-
```
28-
29-
### Run ALL tests (unit + integration)
30-
```bash
31-
cd /Users/rezabarazesh/Documents/test/ci-infra/buildkite/pipeline_generator
32-
python -m pytest tests/ -v
33-
python tests/test_integration_comprehensive.py
34-
```
35-
36-
### Run specific test files
37-
```bash
38-
# Test models
39-
python -m pytest tests/test_models.py -v
40-
41-
# Test transformers
42-
python -m pytest tests/test_transformers.py -v
16+
# All tests
17+
pytest tests/
4318

44-
# Test selection logic
45-
python -m pytest tests/test_selection.py -v
46-
47-
# Test Docker plugin builder
48-
python -m pytest tests/test_docker.py -v
49-
50-
# Test utilities
51-
python -m pytest tests/test_utils.py -v
19+
# Unit tests only
20+
pytest tests/ -k "not integration"
5221

5322
# Integration tests
54-
python -m pytest tests/test_pipeline_generator.py -v
55-
```
56-
57-
### Run with coverage
58-
```bash
59-
python -m pytest tests/ --cov=. --cov-report=html --cov-report=term
60-
```
23+
pytest tests/test_integration_comprehensive.py tests/test_integration_fastcheck.py
6124

62-
### Run specific test class or method
63-
```bash
64-
# Run specific class
65-
python -m pytest tests/test_models.py::TestTestStep -v
25+
# Specific test
26+
pytest tests/test_models.py::TestTestStep
6627

67-
# Run specific test method
68-
python -m pytest tests/test_models.py::TestTestStep::test_create_test_step_with_command -v
28+
# With coverage
29+
pytest tests/ --cov=.. --cov-report=html
6930
```
7031

71-
## Test Structure
72-
73-
```
74-
tests/
75-
├── __init__.py
76-
├── conftest.py # Shared fixtures
77-
├── README.md # This file
78-
├── TEST_SUMMARY.md # Complete test summary
79-
├── run_tests.sh # Unit test runner
80-
│
81-
├── test_integration_comprehensive.py # 56 integration scenarios (backward compatibility)
82-
│
83-
├── test_models.py # Tests for data models
84-
│ ├── TestStepKey # Step key generation
85-
│ ├── TestBlockStep # Block step creation
86-
│ ├── TestTestStep # TestStep model
87-
│ ├── TestBuildkiteStep # BuildkiteStep model
88-
│ └── TestBuildkiteBlockStep # BuildkiteBlockStep model
89-
│
90-
├── test_transformers.py # Tests for command transformers
91-
│ ├── TestNormalizer # Command normalization
92-
│ ├── TestTestTargetingTransformer # Intelligent test targeting
93-
│ └── TestCoverageTransformer # Coverage injection
94-
│
95-
├── test_selection.py # Tests for test selection
96-
│ ├── TestShouldRunStep # Run/skip decisions
97-
│ ├── TestChangedTests # Changed test detection
98-
│ ├── TestIntelligentTestTargeting # Intelligent targeting
99-
│ ├── TestExtractCoveredTestPaths # Test path extraction
100-
│ └── TestShouldBlockStep # Block decisions
101-
│
102-
├── test_docker.py # Tests for Docker plugins
103-
│ ├── TestDockerEnvironment # Environment configuration
104-
│ ├── TestDockerVolumes # Volume configuration
105-
│ ├── TestStandardDockerConfig # Standard Docker config
106-
│ ├── TestSpecialGPUDockerConfig # Special GPU config
107-
│ ├── TestDockerCommandBuilder # Command builder
108-
│ └── TestPluginBuilder # Plugin builder
109-
│
110-
├── test_utils.py # Tests for utilities
111-
│ ├── TestAgentQueue # Agent queue selection
112-
│ ├── TestFullTestCommand # Full command generation
113-
│ ├── TestMultiNodeTestCommand # Multi-node commands
114-
│ └── TestConstants # Constants validation
115-
│
116-
├── test_pipeline_generator.py # Integration tests
117-
│ ├── TestPipelineGeneratorConfig # Config tests
118-
│ ├── TestReadTestSteps # YAML reading
119-
│ ├── TestWriteBuildkitePipeline # YAML writing
120-
│ ├── TestPipelineGenerator # Pipeline generation
121-
│ └── TestEndToEnd # Full integration
122-
│
123-
└── test_files/ # Test data files
124-
├── test-pipeline.yaml # Sample test configuration
125-
└── expected_pipeline.yaml # Expected output
126-
```
32+
## Test Files
12733

128-
## Jinja Compatibility
34+
- `test_models.py` - Data models
35+
- `test_transformers.py` - Command transformations
36+
- `test_selection.py` - Test filtering and blocking
37+
- `test_docker.py` - Docker plugin construction
38+
- `test_utils.py` - Utilities
39+
- `test_pipeline_generator.py` - End-to-end
40+
- `test_integration_comprehensive.py` - 56 CI scenarios
41+
- `test_integration_fastcheck.py` - 8 fastcheck scenarios
12942

130-
The integration test (`test_integration_comprehensive.py`) verifies 100% backward
131-
compatibility with the Jinja template by comparing generated YAMLs.
132-
This ensures that the refactored Python code produces identical output to the original
133-
Jinja template in all cases.
43+
All tests are pytest-based.
13444

0 commit comments

Comments
 (0)