Will using async or bulk endpoints actually improve performance of import processes?Pinterest Buyable Pins Magento IntegrationMagento 2 REST API checkout payment gateway redirectCaching / invalidating cached responses to Magento 2's REST API?Connect Magento site with Php site?Use API rest media management by storeView code (admin)Improve performance on SOAP callsMagento 2 - Routers_Match slow?First steps of creating API integration with Magento2.3Do I really need Magento over WooCommerce

Shift lens vs move body?

Does a Mace of Disruption's Frightened effect override some undead's immunity to the Frightened condition?

Changing JPEG to RAW to use on Lightroom?

Is it ok to record the 'environment' around my workplace?

Biological refrigeration?

Which old Technic set included large yellow motor?

Papers on arXiv solving the same problem at the same time

Is a memoized pure function itself considered pure?

Who was the most successful German spy against Great Britain in WWII, from the contemporary German perspective?

Unlock your Lock

How can I download a file from a host I can only SSH to through another host?

Why is sh (not bash) complaining about functions defined in my .bashrc?

Talk interpreter

Can MuseScore be used programmatically?

Notice period 60 days but I need to join in 45 days

How many birds in the bush?

How does the OS tell whether an "Address is already in use"?

What's the point of fighting monsters in Zelda BoTW?

What's special ammo in Destiny 2?

1mth baby boy keeps peeing through diapers, sometimes diper seems practically unused

Expressing an implication as ILP where each implication term comprises a chain of boolean ORs

LINQ for generating all possible permutations

To what extent should we fear giving offense?

What are the IPSE’s, the ASPE’s, the FRIPSE’s and the GRIPSE’s?



Will using async or bulk endpoints actually improve performance of import processes?


Pinterest Buyable Pins Magento IntegrationMagento 2 REST API checkout payment gateway redirectCaching / invalidating cached responses to Magento 2's REST API?Connect Magento site with Php site?Use API rest media management by storeView code (admin)Improve performance on SOAP callsMagento 2 - Routers_Match slow?First steps of creating API integration with Magento2.3Do I really need Magento over WooCommerce






.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;








0















We have a multi-store setup with 16 stores and every store has it's own language, we have a lot of store-specific information. Most of our products are configurable products with lots of variants, so all in all we have about 6000 simple or configurable products.



We have a middleware in place that manages upload of categories, product and media via Magento2's synchronous REST API. This middleware has its own message queues and concurrency management. I played a bit around and ended up with about 5 concurrent workers as the best tradeoff between performance and Magento-deadlocks.



In the rare case that I have to upload all products, I have to perform about 6000 products * 16 stores = ~ 100.000 API calls which takes a lot of time, eventough our instance is running on an AWS t2.2xlarge instances. Usually it takes more then 24 hours.



I understand how utilizing bulk or asynchronous APIs can facilitate and optimize things when you don't have a message queue in place yet. But do you think switching over to async or bulk APIs could still signifantly optimize performance compared to our solution? I don't want to optimize just the time it takes to hand over a task to Magento but I want to optimize the time it takes a change actually is applied and becomes visible for the customer. If Magento's bulk/async APIs use the exact same internal APIs as the sync-API, I don't see any potential optimizations except saving network latency here. What do you think?



The management interface of our middleware










share|improve this question
























  • Any experiences?

    – derspringendepunk
    Aug 21 at 8:53

















0















We have a multi-store setup with 16 stores and every store has it's own language, we have a lot of store-specific information. Most of our products are configurable products with lots of variants, so all in all we have about 6000 simple or configurable products.



We have a middleware in place that manages upload of categories, product and media via Magento2's synchronous REST API. This middleware has its own message queues and concurrency management. I played a bit around and ended up with about 5 concurrent workers as the best tradeoff between performance and Magento-deadlocks.



In the rare case that I have to upload all products, I have to perform about 6000 products * 16 stores = ~ 100.000 API calls which takes a lot of time, eventough our instance is running on an AWS t2.2xlarge instances. Usually it takes more then 24 hours.



I understand how utilizing bulk or asynchronous APIs can facilitate and optimize things when you don't have a message queue in place yet. But do you think switching over to async or bulk APIs could still signifantly optimize performance compared to our solution? I don't want to optimize just the time it takes to hand over a task to Magento but I want to optimize the time it takes a change actually is applied and becomes visible for the customer. If Magento's bulk/async APIs use the exact same internal APIs as the sync-API, I don't see any potential optimizations except saving network latency here. What do you think?



The management interface of our middleware










share|improve this question
























  • Any experiences?

    – derspringendepunk
    Aug 21 at 8:53













0












0








0








We have a multi-store setup with 16 stores and every store has it's own language, we have a lot of store-specific information. Most of our products are configurable products with lots of variants, so all in all we have about 6000 simple or configurable products.



We have a middleware in place that manages upload of categories, product and media via Magento2's synchronous REST API. This middleware has its own message queues and concurrency management. I played a bit around and ended up with about 5 concurrent workers as the best tradeoff between performance and Magento-deadlocks.



In the rare case that I have to upload all products, I have to perform about 6000 products * 16 stores = ~ 100.000 API calls which takes a lot of time, eventough our instance is running on an AWS t2.2xlarge instances. Usually it takes more then 24 hours.



I understand how utilizing bulk or asynchronous APIs can facilitate and optimize things when you don't have a message queue in place yet. But do you think switching over to async or bulk APIs could still signifantly optimize performance compared to our solution? I don't want to optimize just the time it takes to hand over a task to Magento but I want to optimize the time it takes a change actually is applied and becomes visible for the customer. If Magento's bulk/async APIs use the exact same internal APIs as the sync-API, I don't see any potential optimizations except saving network latency here. What do you think?



The management interface of our middleware










share|improve this question














We have a multi-store setup with 16 stores and every store has it's own language, we have a lot of store-specific information. Most of our products are configurable products with lots of variants, so all in all we have about 6000 simple or configurable products.



We have a middleware in place that manages upload of categories, product and media via Magento2's synchronous REST API. This middleware has its own message queues and concurrency management. I played a bit around and ended up with about 5 concurrent workers as the best tradeoff between performance and Magento-deadlocks.



In the rare case that I have to upload all products, I have to perform about 6000 products * 16 stores = ~ 100.000 API calls which takes a lot of time, eventough our instance is running on an AWS t2.2xlarge instances. Usually it takes more then 24 hours.



I understand how utilizing bulk or asynchronous APIs can facilitate and optimize things when you don't have a message queue in place yet. But do you think switching over to async or bulk APIs could still signifantly optimize performance compared to our solution? I don't want to optimize just the time it takes to hand over a task to Magento but I want to optimize the time it takes a change actually is applied and becomes visible for the customer. If Magento's bulk/async APIs use the exact same internal APIs as the sync-API, I don't see any potential optimizations except saving network latency here. What do you think?



The management interface of our middleware







magento2 api rest






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Aug 14 at 15:02









derspringendepunkderspringendepunk

1




1















  • Any experiences?

    – derspringendepunk
    Aug 21 at 8:53

















  • Any experiences?

    – derspringendepunk
    Aug 21 at 8:53
















Any experiences?

– derspringendepunk
Aug 21 at 8:53





Any experiences?

– derspringendepunk
Aug 21 at 8:53










0






active

oldest

votes














Your Answer








StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "479"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmagento.stackexchange.com%2fquestions%2f285512%2fwill-using-async-or-bulk-endpoints-actually-improve-performance-of-import-proces%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























0






active

oldest

votes








0






active

oldest

votes









active

oldest

votes






active

oldest

votes















draft saved

draft discarded
















































Thanks for contributing an answer to Magento Stack Exchange!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmagento.stackexchange.com%2fquestions%2f285512%2fwill-using-async-or-bulk-endpoints-actually-improve-performance-of-import-proces%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Category:9 (number) SubcategoriesMedia in category "9 (number)"Navigation menuUpload mediaGND ID: 4485639-8Library of Congress authority ID: sh85091979ReasonatorScholiaStatistics

Circuit construction for execution of conditional statements using least significant bitHow are two different registers being used as “control”?How exactly is the stated composite state of the two registers being produced using the $R_zz$ controlled rotations?Efficiently performing controlled rotations in HHLWould this quantum algorithm implementation work?How to prepare a superposed states of odd integers from $1$ to $sqrtN$?Why is this implementation of the order finding algorithm not working?Circuit construction for Hamiltonian simulationHow can I invert the least significant bit of a certain term of a superposed state?Implementing an oracleImplementing a controlled sum operation

Magento 2 “No Payment Methods” in Admin New OrderHow to integrate Paypal Express Checkout with the Magento APIMagento 1.5 - Sales > Order > edit order and shipping methods disappearAuto Invoice Check/Money Order Payment methodAdd more simple payment methods?Shipping methods not showingWhat should I do to change payment methods if changing the configuration has no effects?1.9 - No Payment Methods showing upMy Payment Methods not Showing for downloadable/virtual product when checkout?Magento2 API to access internal payment methodHow to call an existing payment methods in the registration form?