This was the case despite the fact that Gopher is smaller than some ultra-large language software. Gopher has some 280 billion different parameters, or variables that it can tune. That makes it larger than OpenAI’s GPT-3, which has 175 billion. But it is smaller than a system that Microsoft and Nivida collaborated on earlier this year, called Megatron, that has 535 billion, as well as ones constructed by Google, with 1.6 trillion parameters, and Alibaba, with 10 trillion.
Ultra-large language models have
big implications for business: they have already lead to more fluent chatbots and digital assistants, more accurate translation software, better search engines, and programs that can summarize complex documents. DeepMind, however, said it had no plans to commercialize Gopher. “That’s not the focus right now,” said Koray Kavukcuoglo, DeepMind’s vice president of research.
As ultra-large language models are rapidly being commercialized, A.I. researchers and social scientists have raised
ethical concerns about them. Chief among them are concerns that the models often
learn racial, ethnic and gender stereotypes from the texts on which they are trained and that the models are so complex it is impossible to discover and trace these biases prior to deploying the system. In one example, GPT-3, the language A.I. that OpenAI built, often associates Muslims with violent narratives and regurgitates professional gender stereotypes.
