[BUG]: Exporting a year of distributions times out

[awwaiid]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Bank admins reported that when they try to export a whole year of Distributions (CSV export) that they get a timeout (504 error). When they do a smaller time window, like 6 months or less, it exports without issue. We did some timings and found that it is timing out at 60 seconds.

Expected Behavior

Not to error and somehow (either directly or through an email or something) provide the data.

Steps To Reproduce

1. Get a user with a lot of data
2. Go into their distributions, select a large time range (1 year), and export
3. Get an error!

Environment

Criteria for Completion

• User is able to export a large amount of distributions
• Unit / request testing to validate new behavior
• If there is a new asynchronous method (email), document how this works

Anything else?

We should explore:

• Adjusting infrastructure timeouts as a short-term fix (increase timeout)
• Seeing if we can make an estimate of how big the export is and then switch to an email attachment; give the user a toast that says we will email their attachment
• Look into making the export so fast that we don't have to change the UI

For this issue, we are open to exploring all of these.

Code of Conduct

• I've read the Code of Conduct and understand my responsibilities as a member of the Ruby for Good community

[awwaiid]

@janeewheatley looks like we don't have any existing version of asynchronous emailing of a generated CSV. Over in app/mailers/distribution_mailer.rb we do attach a PDF, so that at least shows how to do an attachment.

You'll need to:

• Somewhere decide how many rows are in the current filter
• If it is above some threshold, change the export button to "Email Export" and add a URL parameter like "email=true" or something but still going to /distributions
• In the distributions index, when that parameter is sent, we need to kick off a background job to generate the CSV and export it
• In the distributions index, when that parameter is sent, we should send back the whole html page but with a flash notice that they will be emailed the results
• Create a new job that executes the same code as the distribution index CSV export, but that emails it to the user
• That'll need a new Mailer

maybe something like this as part of the controller (pseudocode):

    if email_csv
      DistributionExportMailerJob.perform_now(
        distributions: @distributions.includes(line_items: :item), organization: current_organization, filters: scope_filters).generate_csv, filename: "Distributions-#{Time.zone.today}.csv"
      )
    end

[github-actions]

This issue is marked as stale due to no activity within 30 days. If no further activity is detected within 7 days, it will be unassigned.

[github-actions]

Automatically unassigned after 7 days of inactivity.

[jparcill]

I can try picking this up: roughly how many rows did the export fail?

[dorner]

The number of rows will depend on how fast your computer is. I guess just keep trying until it gets above 60 seconds. 😁 But actually you should use something closer to 10 seconds, because that's a long time to wait for a save button.

[jparcill]

Opened a PR doing a different approach. Leveraging the rails streaming capabilities to stream the csv to the browser so the download starts immediately

[jparcill]

If we like this approach and this PR gets merged I can create more PRs to migrate the other exports to use streams as well