Artificial intelligence is worse than humans in every way at summarising documents and might actually create additional work for people, a government trial of the technology has found.

Amazon conducted the test earlier this year for Australia’s corporate regulator the Securities and Investments Commission (ASIC) using submissions made to an inquiry. The outcome of the trial was revealed in an answer to a questions on notice at the Senate select committee on adopting artificial intelligence.

The test involved testing generative AI models before selecting one to ingest five submissions from a parliamentary inquiry into audit and consultancy firms. The most promising model, Meta’s open source model Llama2-70B, was prompted to summarise the submissions with a focus on ASIC mentions, recommendations, references to more regulation, and to include the page references and context.

Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts. Then, a group of reviewers blindly assessed the summaries produced by both humans and AI for coherency, length, ASIC references, regulation references and for identifying recommendations. They were unaware that this exercise involved AI at all.

These reviewers overwhelmingly found that the human summaries beat out their AI competitors on every criteria and on every submission, scoring an 81% on an internal rubric compared with the machine’s 47%.

  • ContrarianTrail@lemm.ee
    link
    fedilink
    English
    arrow-up
    20
    arrow-down
    10
    ·
    2 months ago

    Who is claiming that LLMs are generally intelligent? Is it just “they” or can you actually name a company?

    • kautau@lemmy.world
      link
      fedilink
      English
      arrow-up
      11
      arrow-down
      1
      ·
      2 months ago

      I think the idea is that every company is dumping money into LLMs and no other form of alternative AI development to the point that all AI research is LLM based and therefore to investors and those involved, it’s effectively the only only avenue to AGI, though that’s likely not true

    • exanime@lemmy.world
      link
      fedilink
      English
      arrow-up
      10
      arrow-down
      8
      ·
      2 months ago

      You mean the stuff currently peddled everywhere as “Artificial intelligence”?

      Yeah, nobody is saying they are intelligent

      • ContrarianTrail@lemm.ee
        link
        fedilink
        English
        arrow-up
        10
        arrow-down
        1
        ·
        2 months ago

        AI and AGI are not the same thing.

        A chess playing robot is intelligent but it’s so called “narrow intelligence” because it’s really good at one thing but that doesn’t translate to other things. Human are generally intelligent because we can perform a wide range of cognitive tasks. There’s nothing wrong at calling LLM an AI because that’s what it is. I’m not aware of a single AI company claiming to posses an AGI system.

        • exanime@lemmy.world
          link
          fedilink
          English
          arrow-up
          3
          arrow-down
          1
          ·
          2 months ago

          Yes, I missed the implied meaning when you said “generally intelligent”

      • TheGrandNagus@lemmy.world
        link
        fedilink
        English
        arrow-up
        9
        arrow-down
        2
        ·
        2 months ago

        In game NPC actions have been called “AI” for decades. Computers playing chess has been called AI for decades. Lots of stuff has been.

        Nobody thought they were genuinely sentient or sapient.

        The fact that people lumped LLMs, text-to-image generators, machine learning algorithms, image recognition algorithms, etc into a category and called it “AI” doesn’t mean they think it is self aware or intelligent in the way a human would be.

        • exanime@lemmy.world
          link
          fedilink
          English
          arrow-up
          3
          arrow-down
          4
          ·
          2 months ago

          The person I replied to said nobody was claiming LLMs were intelligent. I just posted that the people behind the push for this overhyped bubble are indeed making that claim

          Whether people believe it is something else. But also, many people do believe it

          • Grimy@lemmy.world
            link
            fedilink
            English
            arrow-up
            5
            ·
            2 months ago

            He said generally intelligent, In the context of the first reply using the term AGI. There is a difference between artificial intelligence and artificial general intelligence.

            • exanime@lemmy.world
              link
              fedilink
              English
              arrow-up
              3
              ·
              2 months ago

              I see… At first read I thought the generally was implying somewhat. I missed the meaning in aGi