-1

Im using Spark3 and Im observing that the cached partitions are getting dropped from memory. This is what Im doing:

  1. caching a df
  2. applying a filter on the cached df, assigning the result into a new df and caching it

df1=spark.table("table1").cache() - fraction cached reaches 100% and then drops down df2=df1.filter(col("id).isin(1,2,3,4)).cache()

Have sufficient memory for execution and storage. No spills observed No evictions observed But still after the cached fractions reach 98-100%, start dropping And slowly the percentage picks up as the second df is being cached. It seems like it is recomputing the cache of the first df again.

Why are the cached partitions getting dropped? Why is it recomputing again? Any ideas?

Improve this question
Related questions
3755
How can I create a memory leak in Java?
1172
How to force browsers to reload cached CSS and JS files?
941
How can I measure the actual memory usage of an application or process?
Load 7 more related questions Show fewer related questions

0

Reset to default

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.