Summary: Salesforce Bulk API Usage, though it offers performance benefits over regular APIs is a scarce resource for organizations. It is important to conserve and optimally use the Bulk API.
The Dell Boomi Salesforce connector does not support the Bulk API v2.0 (which essentially makes a determination automatically whether to use the regular or bulk API) at the time of writing. This feature is sought-after in the forums since organizations hit their Bulk API usage limits frequently. Processing much lower volumes of documents through the Bulk API (Bulk API accepts 10,000 documents in a single go) may be a wasted opportunity to use the regular API (Regular accepts 200 document in a batch) which generally has a much higher usage limit and mostly remains underused. A workaround that can mimic the Salesforce Bulk API v2.0 can be built in the lines described below:
Create a Reusable Subprocess that accepts any document shape and makes a determination if the documents need to be sent down a “Regular” vs. “Bulk” path. This is done by a data shape with custom scripting that counts the number of documents and a subsequent decision shape that sends it down a regular path for a count less than, say, 2,000 vs. a bulk path when the count exceeds 2,000. We picked an arbitrary number but experimentally one can determine where the bulk APIs are wasted in their process.
In the Main process add a Flow Control shape to group and process in batches of 10,000 documents.
Duplicate the Salesforce Operation and name the copy with a Prefix or Suffix of Bulk. Ensure to check the Bulk Operation check box and set a batch size of 10,000 (or your process specific optimal number).
Wire the Return paths from the re-usable subprocess to the Bulk vs. Regular API Calls (with any transformations if needed).
This approach may need to be tweaked specifically to the volumes you hit but we hope this explains the general idea. We’ve had great success using this approach to eliminate wastage of Bulk API calls.
This post can also be found in the Boomi Community Forums