Reolving mediawiki runtime and score variations - #158
Closed
marziehlenjaniMeta wants to merge 1 commit into
Closed
Conversation
facebook-github-bot
added
the
CLA Signed
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D77906847 |
marziehlenjaniMeta
force-pushed
the
export-D77906847-to-v2-beta
branch
from
July 10, 2025 21:48
4fd5893 to
ee7c934
Compare
marziehlenjaniMeta
added a commit
to marziehlenjaniMeta/DCPerf
that referenced
this pull request
Jul 10, 2025
Summary: Both Mediawiki and Mediawiki_mini were exhibiting variations in execution time and achieved scores, particularly on ARM machines. Such variations in benchmarking can lead to incorrect conclusions for anyone conducting experiments. Since one of the purposes of DCPerf_mini is to be used in machine health checks, and a key requirement for this is to maintain variation below 3%, it is essential to address this inconsistency. This diff addresses the variation by resolving three sources of randomness: Randomness in Load Generation: By applying a fixed seed, we can control the randomness in load generation. Since the seed affects the score, this diff allows the DCPerf user to configure the seed or use the current time (os.time) as the seed if a negative seed is provided. Random and Prolonged Warmup Times: This issue is resolved by addressing the scenario where overlapping translation frequencies and the duration of each load generated for warmup result in a pending translation every time the server checks the warmup status. By distinguishing when the server is waiting for a pending translation, we can choose to wait rather than initiate a new load. Benchmark Execution Method: Running the benchmark directly from a script, as opposed to executing the command from a subprocess in Python, results in a simpler trace being observed by HHVM. This leads to more efficient translation. Differential Revision: D77906847
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D77906847 |
marziehlenjaniMeta
added a commit
to marziehlenjaniMeta/DCPerf
that referenced
this pull request
Jul 10, 2025
Summary: Pull Request resolved: facebookresearch#158 Both Mediawiki and Mediawiki_mini were exhibiting variations in execution time and achieved scores, particularly on ARM machines. Such variations in benchmarking can lead to incorrect conclusions for anyone conducting experiments. Since one of the purposes of DCPerf_mini is to be used in machine health checks, and a key requirement for this is to maintain variation below 3%, it is essential to address this inconsistency. This diff addresses the variation by resolving three sources of randomness: Randomness in Load Generation: By applying a fixed seed, we can control the randomness in load generation. Since the seed affects the score, this diff allows the DCPerf user to configure the seed or use the current time (os.time) as the seed if a negative seed is provided. Random and Prolonged Warmup Times: This issue is resolved by addressing the scenario where overlapping translation frequencies and the duration of each load generated for warmup result in a pending translation every time the server checks the warmup status. By distinguishing when the server is waiting for a pending translation, we can choose to wait rather than initiate a new load. Benchmark Execution Method: Running the benchmark directly from a script, as opposed to executing the command from a subprocess in Python, results in a simpler trace being observed by HHVM. This leads to more efficient translation. Differential Revision: D77906847
marziehlenjaniMeta
force-pushed
the
export-D77906847-to-v2-beta
branch
from
July 10, 2025 21:54
ee7c934 to
eea0f5b
Compare
marziehlenjaniMeta
added a commit
to marziehlenjaniMeta/DCPerf
that referenced
this pull request
Jul 11, 2025
Summary: Both Mediawiki and Mediawiki_mini were exhibiting variations in execution time and achieved scores, particularly on ARM machines. Such variations in benchmarking can lead to incorrect conclusions for anyone conducting experiments. Since one of the purposes of DCPerf_mini is to be used in machine health checks, and a key requirement for this is to maintain variation below 3%, it is essential to address this inconsistency. This diff addresses the variation by resolving three sources of randomness: Randomness in Load Generation: By applying a fixed seed, we can control the randomness in load generation. Since the seed affects the score, this diff allows the DCPerf user to configure the seed or use the current time (os.time) as the seed if a negative seed is provided. Random and Prolonged Warmup Times: This issue is resolved by addressing the scenario where overlapping translation frequencies and the duration of each load generated for warmup result in a pending translation every time the server checks the warmup status. By distinguishing when the server is waiting for a pending translation, we can choose to wait rather than initiate a new load. Benchmark Execution Method: Running the benchmark directly from a script, as opposed to executing the command from a subprocess in Python, results in a simpler trace being observed by HHVM. This leads to more efficient translation. Differential Revision: D77906847
marziehlenjaniMeta
force-pushed
the
export-D77906847-to-v2-beta
branch
from
July 11, 2025 16:41
eea0f5b to
e66f883
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D77906847 |
marziehlenjaniMeta
force-pushed
the
export-D77906847-to-v2-beta
branch
from
July 11, 2025 16:46
e66f883 to
e92500e
Compare
marziehlenjaniMeta
added a commit
to marziehlenjaniMeta/DCPerf
that referenced
this pull request
Jul 11, 2025
Summary: Pull Request resolved: facebookresearch#158 Both Mediawiki and Mediawiki_mini were exhibiting variations in execution time and achieved scores, particularly on ARM machines. Such variations in benchmarking can lead to incorrect conclusions for anyone conducting experiments. Since one of the purposes of DCPerf_mini is to be used in machine health checks, and a key requirement for this is to maintain variation below 3%, it is essential to address this inconsistency. This diff addresses the variation by resolving three sources of randomness: Randomness in Load Generation: By applying a fixed seed, we can control the randomness in load generation. Since the seed affects the score, this diff allows the DCPerf user to configure the seed or use the current time (os.time) as the seed if a negative seed is provided. Random and Prolonged Warmup Times: This issue is resolved by addressing the scenario where overlapping translation frequencies and the duration of each load generated for warmup result in a pending translation every time the server checks the warmup status. By distinguishing when the server is waiting for a pending translation, we can choose to wait rather than initiate a new load. Benchmark Execution Method: Running the benchmark directly from a script, as opposed to executing the command from a subprocess in Python, results in a simpler trace being observed by HHVM. This leads to more efficient translation. Differential Revision: D77906847
marziehlenjaniMeta
added a commit
to marziehlenjaniMeta/DCPerf
that referenced
this pull request
Jul 11, 2025
Summary: Both Mediawiki and Mediawiki_mini were exhibiting variations in execution time and achieved scores, particularly on ARM machines. Such variations in benchmarking can lead to incorrect conclusions for anyone conducting experiments. Since one of the purposes of DCPerf_mini is to be used in machine health checks, and a key requirement for this is to maintain variation below 3%, it is essential to address this inconsistency. This diff addresses the variation by resolving three sources of randomness: Randomness in Load Generation: By applying a fixed seed, we can control the randomness in load generation. Since the seed affects the score, this diff allows the DCPerf user to configure the seed or use the current time (os.time) as the seed if a negative seed is provided. Random and Prolonged Warmup Times: This issue is resolved by addressing the scenario where overlapping translation frequencies and the duration of each load generated for warmup result in a pending translation every time the server checks the warmup status. By distinguishing when the server is waiting for a pending translation, we can choose to wait rather than initiate a new load. Benchmark Execution Method: Running the benchmark directly from a script, as opposed to executing the command from a subprocess in Python, results in a simpler trace being observed by HHVM. This leads to more efficient translation. Differential Revision: D77906847
marziehlenjaniMeta
force-pushed
the
export-D77906847-to-v2-beta
branch
from
July 11, 2025 21:17
e92500e to
a3a1c3c
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D77906847 |
marziehlenjaniMeta
added a commit
to marziehlenjaniMeta/DCPerf
that referenced
this pull request
Jul 11, 2025
Summary: Pull Request resolved: facebookresearch#158 Both Mediawiki and Mediawiki_mini were exhibiting variations in execution time and achieved scores, particularly on ARM machines. Such variations in benchmarking can lead to incorrect conclusions for anyone conducting experiments. Since one of the purposes of DCPerf_mini is to be used in machine health checks, and a key requirement for this is to maintain variation below 3%, it is essential to address this inconsistency. This diff addresses the variation by resolving three sources of randomness: Randomness in Load Generation: By applying a fixed seed, we can control the randomness in load generation. Since the seed affects the score, this diff allows the DCPerf user to configure the seed or use the current time (os.time) as the seed if a negative seed is provided. Random and Prolonged Warmup Times: This issue is resolved by addressing the scenario where overlapping translation frequencies and the duration of each load generated for warmup result in a pending translation every time the server checks the warmup status. By distinguishing when the server is waiting for a pending translation, we can choose to wait rather than initiate a new load. Benchmark Execution Method: Running the benchmark directly from a script, as opposed to executing the command from a subprocess in Python, results in a simpler trace being observed by HHVM. This leads to more efficient translation. Differential Revision: D77906847
marziehlenjaniMeta
force-pushed
the
export-D77906847-to-v2-beta
branch
from
July 11, 2025 21:22
a3a1c3c to
a19c0ba
Compare
Summary: Both Mediawiki and Mediawiki_mini were exhibiting variations in execution time and achieved scores, particularly on ARM machines. Such variations in benchmarking can lead to incorrect conclusions for anyone conducting experiments. Since one of the purposes of DCPerf_mini is to be used in machine health checks, and a key requirement for this is to maintain variation below 3%, it is essential to address this inconsistency. This diff addresses the variation by resolving three sources of randomness: Randomness in Load Generation: By applying a fixed seed, we can control the randomness in load generation. Since the seed affects the score, this diff allows the DCPerf user to configure the seed or use the current time (os.time) as the seed if a negative seed is provided. Random and Prolonged Warmup Times: This issue is resolved by addressing the scenario where overlapping translation frequencies and the duration of each load generated for warmup result in a pending translation every time the server checks the warmup status. By distinguishing when the server is waiting for a pending translation, we can choose to wait rather than initiate a new load. Benchmark Execution Method: Running the benchmark directly from a script, as opposed to executing the command from a subprocess in Python, results in a simpler trace being observed by HHVM. This leads to more efficient translation. Reviewed By: excelle08 Differential Revision: D77906847
marziehlenjaniMeta
force-pushed
the
export-D77906847-to-v2-beta
branch
from
July 11, 2025 21:27
a19c0ba to
5c18cf5
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D77906847 |
facebook-github-bot
pushed a commit
that referenced
this pull request
Jul 11, 2025
Summary: Pull Request resolved: #158 Both Mediawiki and Mediawiki_mini were exhibiting variations in execution time and achieved scores, particularly on ARM machines. Such variations in benchmarking can lead to incorrect conclusions for anyone conducting experiments. Since one of the purposes of DCPerf_mini is to be used in machine health checks, and a key requirement for this is to maintain variation below 3%, it is essential to address this inconsistency. This diff addresses the variation by resolving three sources of randomness: Randomness in Load Generation: By applying a fixed seed, we can control the randomness in load generation. Since the seed affects the score, this diff allows the DCPerf user to configure the seed or use the current time (os.time) as the seed if a negative seed is provided. Random and Prolonged Warmup Times: This issue is resolved by addressing the scenario where overlapping translation frequencies and the duration of each load generated for warmup result in a pending translation every time the server checks the warmup status. By distinguishing when the server is waiting for a pending translation, we can choose to wait rather than initiate a new load. Benchmark Execution Method: Running the benchmark directly from a script, as opposed to executing the command from a subprocess in Python, results in a simpler trace being observed by HHVM. This leads to more efficient translation. Reviewed By: excelle08 Differential Revision: D77906847 fbshipit-source-id: 4cf92d2691147b73909caa32a1d925f16b39d532
Contributor
|
Close as it's already committed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Both Mediawiki and Mediawiki_mini were exhibiting variations in execution time and achieved scores, particularly on ARM machines. Such variations in benchmarking can lead to incorrect conclusions for anyone conducting experiments. Since one of the purposes of DCPerf_mini is to be used in machine health checks, and a key requirement for this is to maintain variation below 3%, it is essential to address this inconsistency.
This diff addresses the variation by resolving three sources of randomness:
Randomness in Load Generation: By applying a fixed seed, we can control the randomness in load generation. Since the seed affects the score, this diff allows the DCPerf user to configure the seed or use the current time (os.time) as the seed if a negative seed is provided.
Random and Prolonged Warmup Times: This issue is resolved by addressing the scenario where overlapping translation frequencies and the duration of each load generated for warmup result in a pending translation every time the server checks the warmup status. By distinguishing when the server is waiting for a pending translation, we can choose to wait rather than initiate a new load.
Benchmark Execution Method: Running the benchmark directly from a script, as opposed to executing the command from a subprocess in Python, results in a simpler trace being observed by HHVM. This leads to more efficient translation.
Differential Revision: D77906847