[gluon] Refactor gluon test#20
Open
apinge wants to merge 4 commits into
Open
Conversation
Collaborator
Author
|
Still getting the following error with pytest : 0522 update All tests passed. |
sammysun0711
left a comment
sammysun0711
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, leave minor comment.
Comment on lines
+300
to
+310
| ACC_BATCH_SIZES = _acc_batch_sizes() | ||
|
|
||
| # (32, 64): moe_gemm_new.log — B in range(2, 64) all passed; B>=129 mostly AssertionError/OOM. | ||
| # Skip range(128,256), 256/512/768, 2048/8192, 6144+ for now; needs deep dive on mxn_splitk_2s + TILE_N=64. | ||
| ACC_BATCH_SIZES_TILE_64 = list(range(2, 64)) | ||
|
|
||
| _ACC_CASES = ( | ||
| [(32, 128, b) for b in ACC_BATCH_SIZES] | ||
| + [(32, 64, b) for b in ACC_BATCH_SIZES_TILE_64] | ||
| ) | ||
| _ACC_CASE_IDS = [f"TILE={m}x{n}-B={b}" for m, n, b in _ACC_CASES] |
Collaborator
There was a problem hiding this comment.
Can we make it as function like: _acc_batch_sizes, use parameter name that easy to understand.
| _ACC_CASE_IDS = [f"TILE={m}x{n}-B={b}" for m, n, b in _ACC_CASES] | ||
|
|
||
| @pytest.mark.parametrize("HIDDEN_SIZE,INTER_SIZE,TP", [(4096, 1024, 8)]) | ||
| @pytest.mark.parametrize("TILE_M,TILE_N,B", _ACC_CASES, ids=_ACC_CASE_IDS) |
Collaborator
There was a problem hiding this comment.
What is default value for B here?
Comment on lines
+330
to
+334
| # TODO(TILE 32x64): moe_gemm_perf_only.log — mxn_splitk_2s vs ref AssertionError (e.g. B=256): | ||
| # - E=32, INTER_SIZE=2048 FAIL | ||
| # - E=512, INTER_SIZE=1024 FAIL | ||
| # - E=512, INTER_SIZE=2048 FAIL | ||
| # - E=32, INTER_SIZE=1024 PASS (only passing 32x64 combo) |
Collaborator
There was a problem hiding this comment.
Is it limitation of moe_gemm kernel or just missing test case coverage?
Comment on lines
+154
to
+160
| def _acc_batch_sizes(): | ||
| batch = list(range(2, 64)) | ||
| batch += list(range(128, 256)) | ||
| batch += [i * 256 for i in range(1, 4)] | ||
| batch += [i * 2048 for i in range(1, 5)] | ||
| batch += list(range(2048 * 3, 2048 * 3 + 256)) | ||
| entry_common('mxn_splitk_2s', batch=batch, prec=[torch.bfloat16], TILE_M=TILE_M, TILE_N=TILE_N, HIDDEN_SIZE=HIDDEN_SIZE, INTER_SIZE=INTER_SIZE, TP=TP, run_count=0) | ||
| return batch |
Collaborator
There was a problem hiding this comment.
It seem the same function redefine in different moe kernel, can we move it in common place to reuse?
Collaborator
|
I met following issue on gfx942: /root/workspace/pyhiptests/contrib# ./run_gluon_test.sh
=============================================== test session starts ===============================================
platform linux -- Python 3.10.12, pytest-9.0.2, pluggy-1.6.0
rootdir: /root/workspace/pyhip_gluon/pyhip
configfile: pytest.ini
plugins: anyio-4.12.1, asyncio-1.3.0
asyncio: mode=strict, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 979 items
gluon/gemm_splitk.py Fatal Python error: Aborted
Current thread 0x00007f036e483480 (most recent call first):
File "/usr/local/lib/python3.10/dist-packages/triton/backends/amd/compiler.py", line 513 in make_amdgcn
File "/usr/local/lib/python3.10/dist-packages/triton/backends/amd/compiler.py", line 544 in <lambda>
File "/usr/local/lib/python3.10/dist-packages/triton/compiler/compiler.py", line 327 in compile
File "/usr/local/lib/python3.10/dist-packages/triton/runtime/jit.py", line 894 in _do_compile
File "/usr/local/lib/python3.10/dist-packages/triton/runtime/jit.py", line 749 in run
File "/usr/local/lib/python3.10/dist-packages/triton/runtime/jit.py", line 373 in <lambda>
File "/root/workspace/pyhip_test/pyhip_gluon/pyhip/tests/contrib/gluon/gemm_splitk.py", line 109 in run
File "/root/workspace/pyhip_test/pyhip_gluon/pyhip/tests/contrib/gluon/gemm_splitk.py", line 146 in _run_batch
File "/root/workspace/pyhip_test/pyhip_gluon/pyhip/tests/contrib/gluon/gemm_splitk.py", line 193 in entry_common
File "/root/workspace/pyhip_test/pyhip_gluon/pyhip/tests/contrib/gluon/gemm_splitk.py", line 212 in test_acc
File "/usr/local/lib/python3.10/dist-packages/_pytest/python.py", line 166 in pytest_pyfunc_call
File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 121 in _multicall
File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 512 in __call__
File "/usr/local/lib/python3.10/dist-packages/_pytest/python.py", line 1720 in runtest
File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 179 in pytest_runtest_call
File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 121 in _multicall
File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 512 in __call__
File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 245 in <lambda>
File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 353 in from_call
File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 244 in call_and_report
File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 137 in runtestprotocol
File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 118 in pytest_runtest_protocol
File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 121 in _multicall
File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 512 in __call__
File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 396 in pytest_runtestloop
File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 121 in _multicall
File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 512 in __call__
File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 372 in _main
File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 318 in wrap_session
File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 365 in pytest_cmdline_main
File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 121 in _multicall
File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 512 in __call__
File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 199 in main
File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 223 in console_main
File "/usr/local/bin/pytest", line 33 in <module>
Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, Cython.Plex.Actions, Cython.Plex.Scanners, Cython.Compiler.Scanning, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, markupsafe._speedups, hip_utils (total: 69)
./run_gluon_test.sh: line 6: 155604 Aborted (core dumped) pytest $SCRIPT_DIR/gluon |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.