Artificial intelligence is worse than humans in every way at summarising documents and might actually create additional work for people, a government trial of the technology has found.

Amazon conducted the test earlier this year for Australia’s corporate regulator the Securities and Investments Commission (ASIC) using submissions made to an inquiry. The outcome of the trial was revealed in an answer to a questions on notice at the Senate select committee on adopting artificial intelligence.

The test involved testing generative AI models before selecting one to ingest five submissions from a parliamentary inquiry into audit and consultancy firms. The most promising model, Meta’s open source model Llama2-70B, was prompted to summarise the submissions with a focus on ASIC mentions, recommendations, references to more regulation, and to include the page references and context.

Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts. Then, a group of reviewers blindly assessed the summaries produced by both humans and AI for coherency, length, ASIC references, regulation references and for identifying recommendations. They were unaware that this exercise involved AI at all.

These reviewers overwhelmingly found that the human summaries beat out their AI competitors on every criteria and on every submission, scoring an 81% on an internal rubric compared with the machine’s 47%.

  • Grandwolf319@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    2
    ·
    2 months ago

    That’s not what I’m implying. What I’m saying is that wasting time and effort on quality is pointless when the threshold for success is low.

    Yes and my response to that is for some people maybe, for others they don’t want a low threshold, they want few good articles instead of spam of low quality.

    Maybe in other words, engineering tolerances apply to LLMs too. They’re crude devices, but it’s totally fine if you have a crude problem.

    Exactly, I’m saying there is no objective crude problem. You might be okay with simple summaries but I want every single piece of information I consume to have a very high bar.

    • AA5B@lemmy.world
      link
      fedilink
      English
      arrow-up
      2
      ·
      2 months ago

      What if you’re reading Lemmy, and you don’t really feel like reading the article. Is the headline likely to tell you all you need to know or is the ai summary likely to find more info and without the clickbait?

      • Grandwolf319@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        1
        ·
        edit-2
        2 months ago

        Imo it’s on me to either read the article or be okay with not being informed. Don’t get me wrong, a summery is good, but not when it’s not reliable and the article is a click away, some might have a different comfort level.

    • Pennomi@lemmy.world
      link
      fedilink
      English
      arrow-up
      1
      ·
      2 months ago

      Sure, go for it. But good luck paying an army of copywriters to summarize every article you read.