Batch retrieving message attachments

  • 1
  • Question
  • Updated 2 years ago
We are retrieving user message log for 6 month period containing as many as 2,000 records and hundreds of message attachment URIs (mpegs and pdfs).

Our follow-up attempts to retrieve actual message attachments in a loop is causing rate exceed limit errors.

Does the Batch Request GET method support fetching actual binary text files (i.e.in blocks of 50)?

If not, is there another batch operation we can use to overcome rate exceed limit and accelerate the retrieval of large quantities of attachment binaries?
Photo of Automation USA

Automation USA

  • 1,040 Points 1k badge 2x thumb
  • frustrated

Posted 2 years ago

  • 1
Photo of Benjamin Dean

Benjamin Dean

  • 8,602 Points 5k badge 2x thumb
Batch requests are counted as a single API request regardless of the total number of items requested in the batch.

According to our documentation about batch requests (following sentence is URL), the Message Store should support batch requests do. https://developers.ringcentral.com/api-docs/latest/index.html#!#BatchRequests.html

HOWEVER, I do not see this referenced in the section of Message Store about Attachments: https://developers.ringcentral.com/api-docs/latest/index.html#!#RefGetMessageAttachment

You could try by sending a small batch of four or five items and see if it works, but I think you will receive an HTTP 400 response.

Instead I would recommend one of the following:

A: Create additional admin users (service users essentially) and execute multiple instances of this application in server-side code using each of these users to fetch the data.
B: Submit a Developer Support case to request that your Application's Usage Plan be reviewed for increase considerations.



Knowing you can use batch requests to obtain the list, and to get the individual Message Info items, you can pre-fetch all the data you know you'll need to get the attachment information.

Using your numbers from above (and doing some rough math):

2000 records * 300 attachments per record === 600,000 attachments (roughly)

The Basic Usage Plans permit 40 requests per minute: 40 * 60 (1 hour) * 24 (1 day) * 7 (1 week) === 403,200

This means you should be able to fetch all 600,000+ attachments in under two weeks (if you ALWAYS honor the rate limiting and do not get throttled).

Is that not sufficient (considering you have not attempted to download these attachments for at least the past 6 months)?
Photo of Automation USA

Automation USA

  • 1,040 Points 1k badge 2x thumb
Hi Benjamin,

Thanks for your response.

Fortunately not all message records in the message store contain attachments, as a large percentage of them are SMS text messages :)

Actual figures (per user) are approximately 2,000 messages (or about 7 pages of message store records at 300 per page), and of those, roughly 25% (or about 500) of them contain pdf and mpeg attachments. So if my math is correct, it should take about 12-15 minutes to retrieve all 500 attachments.

The use case is for local caching of message store (on device) upon first time login. Following the first mass retrieval each device will continue to update its local message repository differentially. However, thanks to ubiquity, users will sign in and out of multiple devices, and they expect their data to follow them, so message store will most likely need to be re-fetched multiple times (once per device upon first login).

Even if we were to cut off message store caching in half (3 months), that would still mean retrieving about 250 attachments...

We will experiment with your recommendations.
(Edited)
Photo of Benjamin Dean

Benjamin Dean

  • 8,602 Points 5k badge 2x thumb
Let me know how things go and if you need further assistance.
Photo of Automation USA

Automation USA

  • 1,040 Points 1k badge 2x thumb
Thanks Benjamin,

I do have a related question about rate limits. I am told all responses to all endpoints include headers that contain rate limit metadata. However I have never seen such headers in any of the API responses we are getting, nor are they included in the API Explorer when you test endpoints there.

Where can I find these elusive headers?
(Edited)
Photo of Benjamin Dean

Benjamin Dean

  • 8,602 Points 5k badge 2x thumb
I had not noticed that the API Explorer does not show these new response headers. I have created a ticket to remedy this in the future.

There are four (4) different response headers returned by the RingCentral API:
  • X-Rate-Limit-Group (string)
    Provides you with the human-readable name which you can reference in the Developer Portal's Rate Limit section for your application for a particular API resource's limits

  • X-Rate-Limit-LImit (integer, fixed number)
    Although a little redundant, this is the header which defines the actual count of requests which can be made based upon the X-Rate-Limit-Group during the X-Rate-Limit-Window value (in seconds)

  • X-Rate-Limit-Remaining (integer, decremented count)
    This header is an approximate value of the number of calls you have remaining in the current limit-measuring time window

  • X-Rate-Limit-Window (integer, in seconds)
    This header defines the length of time the maximum number of requests (as defined by the X-Rate-Limit-Limit header indicates) can be executed before your requests will be throttled
Photo of Automation USA

Automation USA

  • 1,040 Points 1k badge 2x thumb
Right, I am aware of these response headers, I just have never seen them in any of the API responses that we get for our app.

Our RingCentral responses look identical to the API Explorer responses (no headers).
Photo of Benjamin Dean

Benjamin Dean

  • 8,602 Points 5k badge 2x thumb
Well...that's a problem, isn't it. :)

Could you do me a favor and submit a Developer Support case (https://developers.ringcentral.com/api/support-cases/create) and request that your account is inspected and resolved as to why your API responses are missing the X-Rate-Limit-* headers?
Photo of Automation USA

Automation USA

  • 1,040 Points 1k badge 2x thumb
Will do.

I am going to guess our API responses may be missing the headers for the same/similar reason that your API Explorer is also missing the response headers :)
Photo of John Wang

John Wang, Official Rep

  • 4,840 Points 4k badge 2x thumb
The X-Rate-Limit-* headers are generated for both API calls and API Explorer calls, however, without the proper inspection you will not see them.

While API Explorer API calls will return the X-Rate-Limit-* headers, it seems that the API Explorer UI does not display them which we will look into. You can view the response headers using Google Chrome's Network Inspection of the API Explorer 'account/~/extension/~/message-store' API call as shown in the screen shot below:



Additionally, when I make an API call to 'account/~/extension/~/message-store', I get the following X-Rate-Limit-* headers (lowercased by Ruby) using the Ruby SDK (https://github.com/grokify/ringcentral-sdk-ruby/) using "pp res.headers":

 "x-rate-limit-group"=>"light",
 "x-rate-limit-limit"=>"50",
 "x-rate-limit-remaining"=>"49",
 "x-rate-limit-window"=>"60"

How are you making API calls and to which endpoints where you are not seeing the X-Rate-Limit-* response headers?

If you create a support case, please reference this post.

(Edited)
Photo of Automation USA

Automation USA

  • 1,040 Points 1k badge 2x thumb
We're using php.

I created a support case Friday. Will add this post reference to it.

Support case 04731530
(Edited)
Photo of Automation USA

Automation USA

  • 1,040 Points 1k badge 2x thumb
Benjamin,

Got response headers working (wohoo!). So now at least we know (exactly) where we stand ratewise after each api call.

Thanks
Photo of Automation USA

Automation USA

  • 1,040 Points 1k badge 2x thumb
John,

  1. I understand rate limits are per user per minute. In my opinion, your base rates are still too restrictive, especially for a public app where anticipating usage is difficult to predict. Fetching attachment history is one use case where I believe 40 calls per minute is not going to cut it if you are trying to fetch 100+ attachments. I'd be interested in knowing how your RC Desktop app (or mobile app for that matter) loads message attachments upon first time sign in.
  2. Only type of subscription which might work in our case could be Webhooks. Meantime, we are polling message log every 6 seconds and fetching log entries.
(Edited)
Photo of Benjamin Dean

Benjamin Dean

  • 8,582 Points 5k badge 2x thumb
For the first item, here is some additional information and recommendations:

A) Create a developer support case and request to have your API Rate Limits increased. You would need to provide a very clear and data-driven use-case for us to consider increasing your limits. If you provide enough information and present a strong case, we could review your application's history manually and then if the team approves, could increase your limits if needed.

B) The Desktop app (and mobile apps) are trusted, internally developed applications which have been battle-tested. These applications use a combination of logs and message store lists (to present the UI which indicates there are attachments), and using JIT development download and present the resource when a user clicks on an attachment.

For the second item...
You can subscribe to the message-store events (similar to subscribing for Presence events): https://developers.ringcentral.com/api-docs/latest/index.html#!#RefGetMessagesEvent

By using the subscription to message-store events, you could remove the long-polling scenario.


John may have improvements or other suggestions, but this should help get you moving in the correct direction.
Photo of Automation USA

Automation USA

  • 1,040 Points 1k badge 2x thumb
Hi Benjamin,

Appreciate the help.
A) Support case for Rate Limit increase has already been submitted. I will append additional use-case information in support of my request.

B) I don't believe the RC Desktop app (for example) is fetching attachments "Just In Time" when a user clicks, they are clearly being cached. 
In looking at my own Mac, for instance,  I can see that there are 163 voicemail attachments in my filesystem [~/Library/Application Support/RingCentral/SoftPhone/~{userid}/voicemails/] totaling 11.1 MB. As many as 16 of those voicemails have not been "read", yet they are there, cached.
Likewise, there are 59 fax attachments sitting in the '/faxes' subdirectory, and as may as 6 of them have never been clicked on, but they are there, cached.
Same is the case with profile images in the '/photos' directory.
Looks like a sizable local message store, and there is one of those cache directories in my filesystem for each user I have signed in as...
As call recordings are not available via desktop client, that does not leave much for JIT...
(Edited)
Photo of Benjamin Dean

Benjamin Dean

  • 8,582 Points 5k badge 2x thumb
You may be correct about the Desktop app. I'm not on that team and assumed that is how they would build it, but bootstrapping data during load makes sense as well. Since the desktop app is "internally developed", it is not under the same constraints as any externally developed apps.