[go: nahoru, domu]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak from using tf.constant in loop in TF 2.16 #68196

Open
declanoller opened this issue May 18, 2024 · 3 comments
Open

Memory leak from using tf.constant in loop in TF 2.16 #68196

declanoller opened this issue May 18, 2024 · 3 comments
Assignees
Labels

Comments

@declanoller
Copy link
declanoller commented May 18, 2024

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

No

Source

binary from pip

TensorFlow version

2.16.1

Custom code

Yes

OS platform and distribution

Linux Ubuntu 24.04

Mobile device

No response

Python version

3.12.3

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

I believe I tracked down a memory leak in my code to a tf.constant creation in a short lived object of a class. I can reproduce it in an even simpler way by just creating the constant in a loop. No @tf.function decoration or model training necessary to cause it. It seems to happen if I replace the tf.constant with a tf.random.uniform(). I tried suggestions I've seen elsewhere, like trying to use del on the variable followed by gc.collect().

Some similar looking bug reports:

It looks like upgrading to 2.15 solved it for some people, but it appears to be back in 2.16?

is this expected and I'm doing something wrong?

Standalone code to reproduce the issue

import time
import psutil
import tensorflow as tf


def loop_copy():
    for _ in range(2000):
        y = tf.constant(5.0)  # still leaks with this


start_mem = psutil.Process().memory_info().rss

for _ in range(500):
    time.sleep(0.1)
    print((psutil.Process().memory_info().rss - start_mem) / 10**6)
    loop_copy()


### Relevant log output

```shell
0.0
6.029312
6.815744
7.471104
8.126464
8.781824
9.437184
10.092544
10.747904
11.534336
12.189696
12.845056
13.500416
14.155776
14.811136
15.597568
16.252928
...
@SuryanarayanaY
Copy link
Collaborator

Hi @declanoller ,

I have tested the code in tf-nightly (dev20240520) and found no memory leak. Please refer to attached gist.

@SuryanarayanaY SuryanarayanaY added stat:awaiting response Status - Awaiting response from author type:performance Performance Issue TF 2.16 labels May 22, 2024
@SuryanarayanaY
Copy link
Collaborator

Tested with Tf2.16 also and found no memory leak.Gist for reference.

@google-ml-butler google-ml-butler bot removed the stat:awaiting response Status - Awaiting response from author label May 22, 2024
@declanoller
Copy link
Author
declanoller commented May 22, 2024

@SuryanarayanaY what python version?

I have two machines. Both have TF 2.16.1. However, running the same code on the one with python 3.10 does not leak, while the machine (in the OP) with 3.12 does leak.

edit: I added a print(sys.version) to your gist, it looks like it's using 3.10:

2.16.1
3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]

Can you test it with 3.12 somehow?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants