• supert
    link
    fedilink
    English
    11 year ago

    I can run 4bit quantised llama 70B on a pair of 3090s. Or rent gpu server time. It’s expensive but not prohibitive.

      • supert
        link
        fedilink
        English
        11 year ago

        3k?Can’t recall exactly, and I’m getting hardwarestability issues.

    • @anotherandrew@lemmy.mixdown.ca
      link
      fedilink
      English
      11 year ago

      I’m trying to get to the point where I can locally run a (slow) LLM that I’ve fed my huge ebook collection too and can ask where to find info on $subject, getting title/page info back. The pdfs that are searchable aren’t too bad but finding a way to ocr the older TIFF scan pdfs and getting it to “see” graphs/images are areas I’m stuck on.