Version 1.2.0 issues with GPU

  • Creator
    Topic
  • #55063
    Vladislav KaplanVladislav Kaplan
    Participant
      @vlady

      Hi, Youssef.

      Hope you are doing well. I have some time to compare performance on v1.1.0 vs 1.2.0 on GPU and strange staff happened.

      Before you published 1.2.0 with example to work on GPU I managed to figure out how to do it with v 1.1.0 and I got expected results – 15-30 ms for GPU step_time and 80-90ms for CPU.

      Both GPU/CPU tended to converge accordingly (increase accuracy and decrease loss)

      After upgrading to v1.2.0 more or less same cycle time was demonstrated, but in attached example with GPU activation loss function convergence did not occurs.

      due to the fact that it is just GPU/CPU functionality, I am not sending you VI – it is the same as per example, (just GPU/CPU switch change to visible).

      I also have video taped screenย  – let me know if you’d like to get it – too big to attach to mail.

      Best regards and good luck

      Vlady Kaplan

       

    • Author
      Replies
    • #55064
      Youssef MENJOURYoussef MENJOUR
      Admin
        @youssefmenjour

        Dear Vlady,

        A new version arrives today (1.2.0.8). We are waiting for the end of its creation is which is in progress at the moment (As HAIBAL has more than 4000 VI it takes more than 4 hours to each build to be done …).

        Apologies for the lack of video guide to switch between CPU / GPU (did not have time to do it and it will be done today for future users since you now know how to do)

        Once the new version is released, can you test with it? If the problem persists, we will start looking for this bug. (there will be a lot of changes on this version that should have been called 1.2.1 but it is not important)

        I am preparing a release note for the new version (1.2.0.8).

        Thank you for your feedback !

        #55066
        Youssef MENJOURYoussef MENJOUR
        Admin
          @youssefmenjour

          Correction – The new release will be 1.2.1.1 ๐Ÿ™‚

          #55067
          Vladislav KaplanVladislav Kaplan
          Participant
          Participant
            @vlady

            Need your mail or any mail to send to. Thanks

            #55068
            Vladislav KaplanVladislav Kaplan
            Participant
            Participant
              @vlady

              https://jumbomail.me/j/oNu1e5cdZEqdoJn

              I just would like to notify that for first version GPU was working properly, no issues were detected with convergence.

              Thanks

              Vlady

              #55079
              Youssef MENJOURYoussef MENJOUR
              Admin
                @youssefmenjour

                We found problem. It is a problem of architecture (if you look at your state QMH FIFO you’ll see it). When we change GPU /CPU or CPU /GPU the test thread continu to send FIFO message to Train thread that create this problem.

                We made modification to avoid this problem. In next release MNIST will be updated.

                ๐Ÿ™‚

                #55081
                Youssef MENJOURYoussef MENJOUR
                Admin
                  @youssefmenjour

                  Theoretically this problem did not exist because we were not supposed to play with the architecture. But it’s important that some people play with HAIBAL, we could have run into a bug.

                  MicrosoftTeams-image

                  We had internally a reflexion if you are agree to continu to find bugs or problems we will give you more licence time ๐Ÿ™‚

                  It is important for us to test our system to gain credibility.

                  Thanks Vlady

                • You must be logged in to reply to this topic.