Safely export larger raw data reports than the CSV limit

What is the desired functionality that you are missing?

Today CSV Export of Reports is limited to 1000 rows per default. This limit can be changed per configuration. At the same time, there is a note in the documentation that increasing the limit can lead to an increased usage of memory.

There are at least 2 different ways to achieve this goal

• Graceful Degradation: If the download fails for any reason, then the user is instructed to filter their data down and try again. This allows administrators to set their download limits aggressively high without a terrible user experience
• API with Pagination: The user requests the CSV with a maximum size. If there are still more rows, there's a pagination parameter (e.g. "start=1001")

As it is hard to predict for the administrator how much impact more rows will have on memory since every report data is different and it is unknown how many rows the user wants to export in the end, we should remove this limit and think about building an async or streaming exporter. (OPT-4845)

Which problem are you going to solve with this functionality?

 Most administrators do not change the default config (1000) and often users are not aware of this. This leads to confusion since users find out by accident that data is missing. Additionally, users expect that more data (one customer mentioned they use 2-20k processes) can be exported easily without manual copy-pasting. Today there is no easy way to export more data without changing the configuration setting.

As a result, customers are afraid to increase this limit. On top of this, they don't want to increase the limit because it would involve shutting down the engine.

Hint:

There's a strong trend towards requesting more public APIs on Optimize, and this makes sense to me.