Why does a table with a defined constant in its index compute 10X slower?Why does array packing in Table behave like this?Why is building a table of function values so much slower than just plotting the function?Issue with very large lists in MathematicaWhy does `Table` returns values but `Plot` doesn't plot them?How to speed up computation of medoids?Why is CompilationTarget -> C slower than directly writing with C?ParallelTable much slower than Table on RandomReal with arbitrary precisionPart does not exist in table construction, but I don't explicitly index that highwhy Findroot slower with jacobian?Why does Table[] slow down exponentially with increasing length?
Sorting with IComparable design
Why does splatting create a tuple on the rhs but a list on the lhs?
Need to read my home electrical Meter
Why did Jon Snow admit his fault in S08E06?
Interpretation of ROC AUC score
How would a developer who mostly fixed bugs for years at a company call out their contributions in their CV?
xcolor breaking ligatures
Time complexity of an algorithm: Is it important to state the base of the logarithm?
Why sampling a periodic signal doesn't yield a periodic discrete signal?
Where is Jon going?
What weight should be given to writers groups critiques?
Using too much dialogue?
...And they were stumped for a long time
Final exams: What is the most common protocol for scheduling?
Is superuser the same as root?
Why was this character made Grand Maester?
The disk image is 497GB smaller than the target device
Best shape for a necromancer's undead minions for battle?
Do copyright notices need to be placed at the beginning of a file?
How to melt snow without fire or using body heat?
Burned out due to current job, Can I take a week of vacation between jobs?
Can we assume that a hash function with high collision resistance also means highly uniform distribution?
How to let other coworkers know that I don't share my coworker's political views?
What could a self-sustaining lunar colony slowly lose that would ultimately prove fatal?
Why does a table with a defined constant in its index compute 10X slower?
Why does array packing in Table behave like this?Why is building a table of function values so much slower than just plotting the function?Issue with very large lists in MathematicaWhy does `Table` returns values but `Plot` doesn't plot them?How to speed up computation of medoids?Why is CompilationTarget -> C slower than directly writing with C?ParallelTable much slower than Table on RandomReal with arbitrary precisionPart does not exist in table construction, but I don't explicitly index that highwhy Findroot slower with jacobian?Why does Table[] slow down exponentially with increasing length?
$begingroup$
I need to do some iterative summations. Here is a minimum working example:
data = Table[RandomReal[], x, 1, 1000000];
(* Method 1 *)
Timing[Total[Table[Total[ Table[data[[i]], i, j, 10 + j]], j, 1, Length[data] - 5*10]]]
(* Method 2, with constant index *)
m = 10;
Timing[Total[Table[Total[ Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*m]]]
And here are the outputs:
0.5625, 5.49936*10^6
9.28125, 5.49936*10^6
For some reason, using m=10
makes it much slower. I will need to do a bunch of m
's, so this is the bottom of a larger nest.
What is a faster way to do this?
Late Edit:
Bonus question: How to optimize this one as well:
Timing[Total[Table[ (Total[ Table[data[[i]], i, j, m + j]])^2 , j, 1, Length[data] - 5*m]]]
list-manipulation performance-tuning table
$endgroup$
add a comment |
$begingroup$
I need to do some iterative summations. Here is a minimum working example:
data = Table[RandomReal[], x, 1, 1000000];
(* Method 1 *)
Timing[Total[Table[Total[ Table[data[[i]], i, j, 10 + j]], j, 1, Length[data] - 5*10]]]
(* Method 2, with constant index *)
m = 10;
Timing[Total[Table[Total[ Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*m]]]
And here are the outputs:
0.5625, 5.49936*10^6
9.28125, 5.49936*10^6
For some reason, using m=10
makes it much slower. I will need to do a bunch of m
's, so this is the bottom of a larger nest.
What is a faster way to do this?
Late Edit:
Bonus question: How to optimize this one as well:
Timing[Total[Table[ (Total[ Table[data[[i]], i, j, m + j]])^2 , j, 1, Length[data] - 5*m]]]
list-manipulation performance-tuning table
$endgroup$
add a comment |
$begingroup$
I need to do some iterative summations. Here is a minimum working example:
data = Table[RandomReal[], x, 1, 1000000];
(* Method 1 *)
Timing[Total[Table[Total[ Table[data[[i]], i, j, 10 + j]], j, 1, Length[data] - 5*10]]]
(* Method 2, with constant index *)
m = 10;
Timing[Total[Table[Total[ Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*m]]]
And here are the outputs:
0.5625, 5.49936*10^6
9.28125, 5.49936*10^6
For some reason, using m=10
makes it much slower. I will need to do a bunch of m
's, so this is the bottom of a larger nest.
What is a faster way to do this?
Late Edit:
Bonus question: How to optimize this one as well:
Timing[Total[Table[ (Total[ Table[data[[i]], i, j, m + j]])^2 , j, 1, Length[data] - 5*m]]]
list-manipulation performance-tuning table
$endgroup$
I need to do some iterative summations. Here is a minimum working example:
data = Table[RandomReal[], x, 1, 1000000];
(* Method 1 *)
Timing[Total[Table[Total[ Table[data[[i]], i, j, 10 + j]], j, 1, Length[data] - 5*10]]]
(* Method 2, with constant index *)
m = 10;
Timing[Total[Table[Total[ Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*m]]]
And here are the outputs:
0.5625, 5.49936*10^6
9.28125, 5.49936*10^6
For some reason, using m=10
makes it much slower. I will need to do a bunch of m
's, so this is the bottom of a larger nest.
What is a faster way to do this?
Late Edit:
Bonus question: How to optimize this one as well:
Timing[Total[Table[ (Total[ Table[data[[i]], i, j, m + j]])^2 , j, 1, Length[data] - 5*m]]]
list-manipulation performance-tuning table
list-manipulation performance-tuning table
edited May 16 at 19:53
Carl Woll
79.9k3102207
79.9k3102207
asked May 16 at 17:07
axsvl77axsvl77
425316
425316
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
The problem lies mostly in the inner Table
:
Timing[Total[Table[Total[Table[data[[i]], i, j, 10 + j]], j, 1,Length[data] - 5*10]]]
m = 10;
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*10]]]
0.366407, 5.50276*10^6
8.01738, 5.50276*10^6
I think the reason is this:
Because the global variable m
could theoretically change its value during the computions, the body of the outer table cannot be compiled (without calls to MainEvaluate). At least, the JIT compiler does not analyze the body of the outer loop thoroughly enough to decide that m
won't change.
You can help the JIT compiler by using With
:
With[m = 10,
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1,Length[data] - 5*m]]]
]
0.369601, 5.5049*10^6
Addendum:
By focusing on the post's title, I have completely overlooked the question on how to make it faster. Here is my proposal (c
) vs. the OP's one (a
) and Carl's (b
):
a = With[m = 10,
Total[
Table[Total[Table[data[[i]], i, j, m + j]], j, 1,
Length[data] - 5*m]]
]; // RepeatedTiming // First
b = Total@ListCorrelate[ConstantArray[1., m + 1],
data[[;; -50 + m - 1]]]; // RepeatedTiming // First
c = Plus[
Range[1., m].data[[1 ;; m]],
(m + 1) Total[data[[m + 1 ;; -5*m - 1]]],
Range[N@m, 1., -1].data[[-5 m ;; -4 m - 1]]
]; // RepeatedTiming // First
a == b == c
0.28
0.017
0.0018
True
$endgroup$
add a comment |
$begingroup$
You can use ListCorrelate
:
m=10;
Total @ ListCorrelate[ConstantArray[1,m+1], data[[;;-4 m-1]]] //AbsoluteTiming
0.017725, 5.50044*10^6
Bonus question
For the bonus question:
data = RandomReal[1, 10^5];
Your version:
With[m = 10,
Total[Table[(Total[Table[data[[i]],i,j,m+j]])^2,j,1,Length[data]-5*m]]
] //AbsoluteTiming
0.448739, 3.11778*10^7
Using ListCorrelate
again:
m = 10;
#.#& @ ListCorrelate[ConstantArray[1, m+1], data[[ ;; -4 m - 1]]] //AbsoluteTiming
0.018401, 3.11778*10^7
$endgroup$
$begingroup$
Carl, nice use of the dot-product to overcome the internal compiling issues that would prevent implementing the mapping of a slotted set! I was attempting a solution with this same thought process, but was unable to overcome the change of #^2 to #1^2 during my attempts at a solution. Your use of ListCorrelate will be helpful in the future, thank you!
$endgroup$
– CA Trevillian
May 17 at 14:21
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "387"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f198514%2fwhy-does-a-table-with-a-defined-constant-in-its-index-compute-10x-slower%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
The problem lies mostly in the inner Table
:
Timing[Total[Table[Total[Table[data[[i]], i, j, 10 + j]], j, 1,Length[data] - 5*10]]]
m = 10;
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*10]]]
0.366407, 5.50276*10^6
8.01738, 5.50276*10^6
I think the reason is this:
Because the global variable m
could theoretically change its value during the computions, the body of the outer table cannot be compiled (without calls to MainEvaluate). At least, the JIT compiler does not analyze the body of the outer loop thoroughly enough to decide that m
won't change.
You can help the JIT compiler by using With
:
With[m = 10,
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1,Length[data] - 5*m]]]
]
0.369601, 5.5049*10^6
Addendum:
By focusing on the post's title, I have completely overlooked the question on how to make it faster. Here is my proposal (c
) vs. the OP's one (a
) and Carl's (b
):
a = With[m = 10,
Total[
Table[Total[Table[data[[i]], i, j, m + j]], j, 1,
Length[data] - 5*m]]
]; // RepeatedTiming // First
b = Total@ListCorrelate[ConstantArray[1., m + 1],
data[[;; -50 + m - 1]]]; // RepeatedTiming // First
c = Plus[
Range[1., m].data[[1 ;; m]],
(m + 1) Total[data[[m + 1 ;; -5*m - 1]]],
Range[N@m, 1., -1].data[[-5 m ;; -4 m - 1]]
]; // RepeatedTiming // First
a == b == c
0.28
0.017
0.0018
True
$endgroup$
add a comment |
$begingroup$
The problem lies mostly in the inner Table
:
Timing[Total[Table[Total[Table[data[[i]], i, j, 10 + j]], j, 1,Length[data] - 5*10]]]
m = 10;
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*10]]]
0.366407, 5.50276*10^6
8.01738, 5.50276*10^6
I think the reason is this:
Because the global variable m
could theoretically change its value during the computions, the body of the outer table cannot be compiled (without calls to MainEvaluate). At least, the JIT compiler does not analyze the body of the outer loop thoroughly enough to decide that m
won't change.
You can help the JIT compiler by using With
:
With[m = 10,
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1,Length[data] - 5*m]]]
]
0.369601, 5.5049*10^6
Addendum:
By focusing on the post's title, I have completely overlooked the question on how to make it faster. Here is my proposal (c
) vs. the OP's one (a
) and Carl's (b
):
a = With[m = 10,
Total[
Table[Total[Table[data[[i]], i, j, m + j]], j, 1,
Length[data] - 5*m]]
]; // RepeatedTiming // First
b = Total@ListCorrelate[ConstantArray[1., m + 1],
data[[;; -50 + m - 1]]]; // RepeatedTiming // First
c = Plus[
Range[1., m].data[[1 ;; m]],
(m + 1) Total[data[[m + 1 ;; -5*m - 1]]],
Range[N@m, 1., -1].data[[-5 m ;; -4 m - 1]]
]; // RepeatedTiming // First
a == b == c
0.28
0.017
0.0018
True
$endgroup$
add a comment |
$begingroup$
The problem lies mostly in the inner Table
:
Timing[Total[Table[Total[Table[data[[i]], i, j, 10 + j]], j, 1,Length[data] - 5*10]]]
m = 10;
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*10]]]
0.366407, 5.50276*10^6
8.01738, 5.50276*10^6
I think the reason is this:
Because the global variable m
could theoretically change its value during the computions, the body of the outer table cannot be compiled (without calls to MainEvaluate). At least, the JIT compiler does not analyze the body of the outer loop thoroughly enough to decide that m
won't change.
You can help the JIT compiler by using With
:
With[m = 10,
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1,Length[data] - 5*m]]]
]
0.369601, 5.5049*10^6
Addendum:
By focusing on the post's title, I have completely overlooked the question on how to make it faster. Here is my proposal (c
) vs. the OP's one (a
) and Carl's (b
):
a = With[m = 10,
Total[
Table[Total[Table[data[[i]], i, j, m + j]], j, 1,
Length[data] - 5*m]]
]; // RepeatedTiming // First
b = Total@ListCorrelate[ConstantArray[1., m + 1],
data[[;; -50 + m - 1]]]; // RepeatedTiming // First
c = Plus[
Range[1., m].data[[1 ;; m]],
(m + 1) Total[data[[m + 1 ;; -5*m - 1]]],
Range[N@m, 1., -1].data[[-5 m ;; -4 m - 1]]
]; // RepeatedTiming // First
a == b == c
0.28
0.017
0.0018
True
$endgroup$
The problem lies mostly in the inner Table
:
Timing[Total[Table[Total[Table[data[[i]], i, j, 10 + j]], j, 1,Length[data] - 5*10]]]
m = 10;
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*10]]]
0.366407, 5.50276*10^6
8.01738, 5.50276*10^6
I think the reason is this:
Because the global variable m
could theoretically change its value during the computions, the body of the outer table cannot be compiled (without calls to MainEvaluate). At least, the JIT compiler does not analyze the body of the outer loop thoroughly enough to decide that m
won't change.
You can help the JIT compiler by using With
:
With[m = 10,
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1,Length[data] - 5*m]]]
]
0.369601, 5.5049*10^6
Addendum:
By focusing on the post's title, I have completely overlooked the question on how to make it faster. Here is my proposal (c
) vs. the OP's one (a
) and Carl's (b
):
a = With[m = 10,
Total[
Table[Total[Table[data[[i]], i, j, m + j]], j, 1,
Length[data] - 5*m]]
]; // RepeatedTiming // First
b = Total@ListCorrelate[ConstantArray[1., m + 1],
data[[;; -50 + m - 1]]]; // RepeatedTiming // First
c = Plus[
Range[1., m].data[[1 ;; m]],
(m + 1) Total[data[[m + 1 ;; -5*m - 1]]],
Range[N@m, 1., -1].data[[-5 m ;; -4 m - 1]]
]; // RepeatedTiming // First
a == b == c
0.28
0.017
0.0018
True
edited May 16 at 19:12
answered May 16 at 17:15
Henrik SchumacherHenrik Schumacher
63k587176
63k587176
add a comment |
add a comment |
$begingroup$
You can use ListCorrelate
:
m=10;
Total @ ListCorrelate[ConstantArray[1,m+1], data[[;;-4 m-1]]] //AbsoluteTiming
0.017725, 5.50044*10^6
Bonus question
For the bonus question:
data = RandomReal[1, 10^5];
Your version:
With[m = 10,
Total[Table[(Total[Table[data[[i]],i,j,m+j]])^2,j,1,Length[data]-5*m]]
] //AbsoluteTiming
0.448739, 3.11778*10^7
Using ListCorrelate
again:
m = 10;
#.#& @ ListCorrelate[ConstantArray[1, m+1], data[[ ;; -4 m - 1]]] //AbsoluteTiming
0.018401, 3.11778*10^7
$endgroup$
$begingroup$
Carl, nice use of the dot-product to overcome the internal compiling issues that would prevent implementing the mapping of a slotted set! I was attempting a solution with this same thought process, but was unable to overcome the change of #^2 to #1^2 during my attempts at a solution. Your use of ListCorrelate will be helpful in the future, thank you!
$endgroup$
– CA Trevillian
May 17 at 14:21
add a comment |
$begingroup$
You can use ListCorrelate
:
m=10;
Total @ ListCorrelate[ConstantArray[1,m+1], data[[;;-4 m-1]]] //AbsoluteTiming
0.017725, 5.50044*10^6
Bonus question
For the bonus question:
data = RandomReal[1, 10^5];
Your version:
With[m = 10,
Total[Table[(Total[Table[data[[i]],i,j,m+j]])^2,j,1,Length[data]-5*m]]
] //AbsoluteTiming
0.448739, 3.11778*10^7
Using ListCorrelate
again:
m = 10;
#.#& @ ListCorrelate[ConstantArray[1, m+1], data[[ ;; -4 m - 1]]] //AbsoluteTiming
0.018401, 3.11778*10^7
$endgroup$
$begingroup$
Carl, nice use of the dot-product to overcome the internal compiling issues that would prevent implementing the mapping of a slotted set! I was attempting a solution with this same thought process, but was unable to overcome the change of #^2 to #1^2 during my attempts at a solution. Your use of ListCorrelate will be helpful in the future, thank you!
$endgroup$
– CA Trevillian
May 17 at 14:21
add a comment |
$begingroup$
You can use ListCorrelate
:
m=10;
Total @ ListCorrelate[ConstantArray[1,m+1], data[[;;-4 m-1]]] //AbsoluteTiming
0.017725, 5.50044*10^6
Bonus question
For the bonus question:
data = RandomReal[1, 10^5];
Your version:
With[m = 10,
Total[Table[(Total[Table[data[[i]],i,j,m+j]])^2,j,1,Length[data]-5*m]]
] //AbsoluteTiming
0.448739, 3.11778*10^7
Using ListCorrelate
again:
m = 10;
#.#& @ ListCorrelate[ConstantArray[1, m+1], data[[ ;; -4 m - 1]]] //AbsoluteTiming
0.018401, 3.11778*10^7
$endgroup$
You can use ListCorrelate
:
m=10;
Total @ ListCorrelate[ConstantArray[1,m+1], data[[;;-4 m-1]]] //AbsoluteTiming
0.017725, 5.50044*10^6
Bonus question
For the bonus question:
data = RandomReal[1, 10^5];
Your version:
With[m = 10,
Total[Table[(Total[Table[data[[i]],i,j,m+j]])^2,j,1,Length[data]-5*m]]
] //AbsoluteTiming
0.448739, 3.11778*10^7
Using ListCorrelate
again:
m = 10;
#.#& @ ListCorrelate[ConstantArray[1, m+1], data[[ ;; -4 m - 1]]] //AbsoluteTiming
0.018401, 3.11778*10^7
edited May 16 at 20:04
answered May 16 at 17:21
Carl WollCarl Woll
79.9k3102207
79.9k3102207
$begingroup$
Carl, nice use of the dot-product to overcome the internal compiling issues that would prevent implementing the mapping of a slotted set! I was attempting a solution with this same thought process, but was unable to overcome the change of #^2 to #1^2 during my attempts at a solution. Your use of ListCorrelate will be helpful in the future, thank you!
$endgroup$
– CA Trevillian
May 17 at 14:21
add a comment |
$begingroup$
Carl, nice use of the dot-product to overcome the internal compiling issues that would prevent implementing the mapping of a slotted set! I was attempting a solution with this same thought process, but was unable to overcome the change of #^2 to #1^2 during my attempts at a solution. Your use of ListCorrelate will be helpful in the future, thank you!
$endgroup$
– CA Trevillian
May 17 at 14:21
$begingroup$
Carl, nice use of the dot-product to overcome the internal compiling issues that would prevent implementing the mapping of a slotted set! I was attempting a solution with this same thought process, but was unable to overcome the change of #^2 to #1^2 during my attempts at a solution. Your use of ListCorrelate will be helpful in the future, thank you!
$endgroup$
– CA Trevillian
May 17 at 14:21
$begingroup$
Carl, nice use of the dot-product to overcome the internal compiling issues that would prevent implementing the mapping of a slotted set! I was attempting a solution with this same thought process, but was unable to overcome the change of #^2 to #1^2 during my attempts at a solution. Your use of ListCorrelate will be helpful in the future, thank you!
$endgroup$
– CA Trevillian
May 17 at 14:21
add a comment |
Thanks for contributing an answer to Mathematica Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f198514%2fwhy-does-a-table-with-a-defined-constant-in-its-index-compute-10x-slower%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown